[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225929848825-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:58:49 dist-test-slave-70st krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:58:49 dist-test-slave-70st krb5kdc[399](info): set up 2 sockets
Sep 24 04:58:49 dist-test-slave-70st krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260924 04:58:50.797753 392 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225929848825-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225929848825-392-0/krb5kdc/test-user.keytab.
Sep 24 04:58:50 dist-test-slave-70st krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225930, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:58:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:58:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:58:50.904224 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225929848825-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225929848825-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225929848825-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35327
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35327
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:58:50.982535 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:58:50.982729 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:58:50.982749 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:58:50.983834 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:58:50.983873 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:58:50.983886 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:58:50.983897 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:58:50.983908 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:58:50.985299 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225929848825-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225929848825-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35327
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35327
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225929848825-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 04:58:50.985579 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:58:50.985790 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:58:50.990510 412 server_base.cc:1034] running on GCE node
W20260924 04:58:50.990437 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:58:50.990437 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:58:50.990479 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:58:50.990949 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:58:50.991219 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:58:50.992393 412 hybrid_clock.cc:648] HybridClock initialized: now 1790225930992341 us; error 41 us; skew 500 ppm
Sep 24 04:58:50 dist-test-slave-70st krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225930, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:58:50.994771 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:58:50.995217 412 webserver.cc:460] Webserver started at http://127.0.98.62:39935/ using document root <none> and password file <none>
I20260924 04:58:50.995431 412 fs_manager.cc:359] Metadata directory not provided
I20260924 04:58:50.995474 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:58:50.995594 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:58:50.996639 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225929848825-392-0/minicluster-data/master-0/data/instance:
uuid: "4e2200236ef3470084998badc33f4cc5"
format_stamp: "Formatted at 2026-09-24 04:58:50 on dist-test-slave-70st"
server_key: "841cc6a5cdb74233986ef0cb6c6e0aae"
server_key_iv: "5b49833d099c83eae5381bb363d96922"
server_key_version: "encryptionkey@0"
I20260924 04:58:50.996953 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225929848825-392-0/minicluster-data/master-0/wal/instance:
uuid: "4e2200236ef3470084998badc33f4cc5"
format_stamp: "Formatted at 2026-09-24 04:58:50 on dist-test-slave-70st"
server_key: "841cc6a5cdb74233986ef0cb6c6e0aae"
server_key_iv: "5b49833d099c83eae5381bb363d96922"
server_key_version: "encryptionkey@0"
I20260924 04:58:50.998224 412 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 04:58:50.998883 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:58:50.999027 412 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 04:58:50.999099 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225929848825-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225929848825-392-0/minicluster-data/master-0/wal
uuid: "4e2200236ef3470084998badc33f4cc5"
format_stamp: "Formatted at 2026-09-24 04:58:50 on dist-test-slave-70st"
server_key: "841cc6a5cdb74233986ef0cb6c6e0aae"
server_key_iv: "5b49833d099c83eae5381bb363d96922"
server_key_version: "encryptionkey@0"
I20260924 04:58:50.999161 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225929848825-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225929848825-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225929848825-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:58:51.027101 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:58:51.037688 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:58:51.037824 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:58:51.041599 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35327
I20260924 04:58:51.041646 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35327 every 8 connection(s)
I20260924 04:58:51.041965 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:58:51.042822 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:58:51.045060 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e2200236ef3470084998badc33f4cc5: Bootstrap starting.
I20260924 04:58:51.045650 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4e2200236ef3470084998badc33f4cc5: Neither blocks nor log segments found. Creating new log.
I20260924 04:58:51.045899 480 log.cc:824] T 00000000000000000000000000000000 P 4e2200236ef3470084998badc33f4cc5: Log is configured to *not* fsync() on all Append() calls
I20260924 04:58:51.046567 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e2200236ef3470084998badc33f4cc5: No bootstrap required, opened a new log
I20260924 04:58:51.048056 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e2200236ef3470084998badc33f4cc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e2200236ef3470084998badc33f4cc5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35327 } }
I20260924 04:58:51.048188 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4e2200236ef3470084998badc33f4cc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:58:51.048220 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e2200236ef3470084998badc33f4cc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e2200236ef3470084998badc33f4cc5, State: Initialized, Role: FOLLOWER
I20260924 04:58:51.048375 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e2200236ef3470084998badc33f4cc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e2200236ef3470084998badc33f4cc5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35327 } }
I20260924 04:58:51.048458 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4e2200236ef3470084998badc33f4cc5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:58:51.048496 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4e2200236ef3470084998badc33f4cc5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:58:51.048540 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e2200236ef3470084998badc33f4cc5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:58:51.048691 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 412
I20260924 04:58:51.048790 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225929848825-392-0/minicluster-data/master-0/wal/instance
I20260924 04:58:51.049187 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4e2200236ef3470084998badc33f4cc5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e2200236ef3470084998badc33f4cc5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35327 } }
I20260924 04:58:51.049288 480 leader_election.cc:302] T 00000000000000000000000000000000 P 4e2200236ef3470084998badc33f4cc5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4e2200236ef3470084998badc33f4cc5; no voters:
I20260924 04:58:51.049482 480 leader_election.cc:288] T 00000000000000000000000000000000 P 4e2200236ef3470084998badc33f4cc5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:58:51.049511 483 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4e2200236ef3470084998badc33f4cc5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:58:51.049604 392 external_mini_cluster.cc:1428] Setting key ae36ec8fe79d6819b244dae146442084
I20260924 04:58:51.049642 483 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4e2200236ef3470084998badc33f4cc5 [term 1 LEADER]: Becoming Leader. State: Replica: 4e2200236ef3470084998badc33f4cc5, State: Running, Role: LEADER
I20260924 04:58:51.049718 483 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e2200236ef3470084998badc33f4cc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e2200236ef3470084998badc33f4cc5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35327 } }
I20260924 04:58:51.049741 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e2200236ef3470084998badc33f4cc5 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:58:51.050071 483 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e2200236ef3470084998badc33f4cc5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e2200236ef3470084998badc33f4cc5. Latest consensus state: current_term: 1 leader_uuid: "4e2200236ef3470084998badc33f4cc5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e2200236ef3470084998badc33f4cc5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35327 } } }
I20260924 04:58:51.050141 483 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e2200236ef3470084998badc33f4cc5 [sys.catalog]: This master's current role is: LEADER
I20260924 04:58:51.050880 488 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:58:51.050962 485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e2200236ef3470084998badc33f4cc5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4e2200236ef3470084998badc33f4cc5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e2200236ef3470084998badc33f4cc5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35327 } } }
I20260924 04:58:51.051047 485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e2200236ef3470084998badc33f4cc5 [sys.catalog]: This master's current role is: LEADER
I20260924 04:58:51.051396 488 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:58:51.052958 488 catalog_manager.cc:1347] Generated new cluster ID: a9047e8367a94cf6953d462635726b0d
I20260924 04:58:51.053022 488 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 04:58:51 dist-test-slave-70st krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225930, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:58:51.064218 502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:58:51.051998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43048 (local address 127.0.98.62:35327)
0924 04:58:51.052144 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 04:58:51.052152 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 04:58:51.052183 (+ 31us) server_negotiation.cc:407] Connection header received
0924 04:58:51.053842 (+ 1659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:58:51.053849 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:58:51.053934 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:58:51.054081 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:58:51.055526 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:58:51.056428 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:58:51.057392 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:58:51.057828 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:58:51.057926 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:58:51.061025 (+ 3099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:58:51.061028 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:58:51.061028 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:58:51.061037 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:58:51.062535 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:58:51.062916 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:58:51.062919 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:58:51.062920 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:58:51.062996 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:58:51.063209 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:58:51.063210 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:58:51.063210 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:58:51.063433 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:58:51.063580 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 04:58:51.063840 (+ 260us) server_negotiation.cc:299] Negotiation successful
0924 04:58:51.063989 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":64,"threads_started":1}
I20260924 04:58:51.064306 488 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:58:51.064926 488 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:58:51.070430 488 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4e2200236ef3470084998badc33f4cc5: Generated new TSK 0
I20260924 04:58:51.070583 488 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225929848825-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225929848825-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:58:51.096429 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225929848825-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225929848825-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225929848825-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225929848825-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35327
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225929848825-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:58:51.174029 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:58:51.174232 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:58:51.174263 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:58:51.175429 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:58:51.175472 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:58:51.175506 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:58:51.176940 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225929848825-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225929848825-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225929848825-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225929848825-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35327
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225929848825-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 04:58:51.177175 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:58:51.177385 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:58:51.182262 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:58:51.182262 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:58:51.182399 506 server_base.cc:1034] running on GCE node
W20260924 04:58:51.182266 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:58:51.182677 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:58:51.182893 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:58:51.184026 506 hybrid_clock.cc:648] HybridClock initialized: now 1790225931184018 us; error 26 us; skew 500 ppm
Sep 24 04:58:51 dist-test-slave-70st krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225931, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:58:51.186388 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:58:51.186755 506 webserver.cc:460] Webserver started at http://127.0.98.1:36167/ using document root <none> and password file <none>
I20260924 04:58:51.186954 506 fs_manager.cc:359] Metadata directory not provided
I20260924 04:58:51.186995 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:58:51.187108 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:58:51.188104 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225929848825-392-0/minicluster-data/ts-0/data/instance:
uuid: "dfcdb867145f47f2bba4d2fa0f74a86c"
format_stamp: "Formatted at 2026-09-24 04:58:51 on dist-test-slave-70st"
server_key: "a2711ee7334dea46e58fda690c495649"
server_key_iv: "11b875717daf8cf48017a19ad76caeca"
server_key_version: "encryptionkey@0"
I20260924 04:58:51.188441 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225929848825-392-0/minicluster-data/ts-0/wal/instance:
uuid: "dfcdb867145f47f2bba4d2fa0f74a86c"
format_stamp: "Formatted at 2026-09-24 04:58:51 on dist-test-slave-70st"
server_key: "a2711ee7334dea46e58fda690c495649"
server_key_iv: "11b875717daf8cf48017a19ad76caeca"
server_key_version: "encryptionkey@0"
I20260924 04:58:51.189652 506 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 04:58:51.190301 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:58:51.190459 506 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 04:58:51.190541 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225929848825-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225929848825-392-0/minicluster-data/ts-0/wal
uuid: "dfcdb867145f47f2bba4d2fa0f74a86c"
format_stamp: "Formatted at 2026-09-24 04:58:51 on dist-test-slave-70st"
server_key: "a2711ee7334dea46e58fda690c495649"
server_key_iv: "11b875717daf8cf48017a19ad76caeca"
server_key_version: "encryptionkey@0"
I20260924 04:58:51.190603 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225929848825-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225929848825-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225929848825-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:58:51.207760 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:58:51.209914 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:58:51.210043 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:58:51.210260 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:58:51.210562 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:58:51.210595 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:58:51.210619 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:58:51.210642 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:58:51.216264 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43453
I20260924 04:58:51.216372 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43453 every 8 connection(s)
I20260924 04:58:51.216764 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225929848825-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:58:51.220937 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 506
I20260924 04:58:51.221022 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225929848825-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:58:51.221161 392 external_mini_cluster.cc:1428] Setting key 885b34cd1967c06ccfa5f04326637c63
Sep 24 04:58:51 dist-test-slave-70st krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225931, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:58:51.225777 502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:58:51.217703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36763 (local address 127.0.98.62:35327)
0924 04:58:51.217755 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 04:58:51.217757 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:58:51.217901 (+ 144us) server_negotiation.cc:407] Connection header received
0924 04:58:51.218367 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:58:51.218368 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:58:51.218380 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:58:51.218450 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:58:51.219290 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:58:51.219788 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:58:51.220785 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:58:51.220961 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:58:51.221010 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:58:51.223253 (+ 2243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:58:51.223255 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:58:51.223255 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:58:51.223262 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:58:51.224477 (+ 1215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:58:51.224858 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:58:51.224860 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:58:51.224860 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:58:51.224917 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:58:51.225135 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:58:51.225136 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:58:51.225136 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:58:51.225244 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:58:51.225299 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 04:58:51.225599 (+ 300us) server_negotiation.cc:299] Negotiation successful
0924 04:58:51.225686 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 04:58:51.225968 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:58:51.217648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35327 (local address 127.0.98.1:36763)
0924 04:58:51.217797 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 04:58:51.217839 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 04:58:51.218286 (+ 447us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:58:51.218493 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:58:51.218496 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:58:51.219027 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:58:51.219221 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:58:51.219225 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:58:51.219867 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:58:51.219868 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:58:51.220709 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:58:51.220711 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:58:51.221032 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:58:51.221033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:58:51.221178 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:58:51.221711 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:58:51.221720 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:58:51.223160 (+ 1440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:58:51.224551 (+ 1391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:58:51.224557 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:58:51.224559 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:58:51.224793 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:58:51.224970 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:58:51.224971 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:58:51.224971 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:58:51.225079 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:58:51.225298 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:58:51.225304 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:58:51.225472 (+ 168us) client_negotiation.cc:769] Sending connection context
0924 04:58:51.225571 (+ 99us) client_negotiation.cc:240] Negotiation successful
0924 04:58:51.225722 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":64,"threads_started":1}
I20260924 04:58:51.226416 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35327
I20260924 04:58:51.226503 635 heartbeater.cc:464] Registering TS with master...
I20260924 04:58:51.226670 635 heartbeater.cc:511] Master 127.0.98.62:35327 requested a full tablet report, sending...
I20260924 04:58:51.227239 444 ts_manager.cc:194] Registered new tserver with Master: dfcdb867145f47f2bba4d2fa0f74a86c (127.0.98.1:43453)
I20260924 04:58:51.228163 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:36763
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225929848825-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225929848825-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:58:51.244488 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225929848825-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225929848825-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225929848825-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225929848825-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35327
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225929848825-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:58:51.321503 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:58:51.321692 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:58:51.321714 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:58:51.322818 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:58:51.322857 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:58:51.322885 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:58:51.324254 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225929848825-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225929848825-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225929848825-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225929848825-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35327
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225929848825-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 04:58:51.324461 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:58:51.324661 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:58:51.329242 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:58:51.329249 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:58:51.329241 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:58:51.329273 640 server_base.cc:1034] running on GCE node
I20260924 04:58:51.329643 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:58:51.329876 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:58:51.331022 640 hybrid_clock.cc:648] HybridClock initialized: now 1790225931331011 us; error 34 us; skew 500 ppm
Sep 24 04:58:51 dist-test-slave-70st krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225931, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:58:51.333181 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:58:51.333632 640 webserver.cc:460] Webserver started at http://127.0.98.2:35555/ using document root <none> and password file <none>
I20260924 04:58:51.333834 640 fs_manager.cc:359] Metadata directory not provided
I20260924 04:58:51.333879 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:58:51.334002 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:58:51.335000 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225929848825-392-0/minicluster-data/ts-1/data/instance:
uuid: "0f34bf2209e94c938c7ceea1a335da2b"
format_stamp: "Formatted at 2026-09-24 04:58:51 on dist-test-slave-70st"
server_key: "a49c6d1afc4dead2b402bb63d1e0ae22"
server_key_iv: "e17488ba708b0bf06cb753265866c484"
server_key_version: "encryptionkey@0"
I20260924 04:58:51.335326 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225929848825-392-0/minicluster-data/ts-1/wal/instance:
uuid: "0f34bf2209e94c938c7ceea1a335da2b"
format_stamp: "Formatted at 2026-09-24 04:58:51 on dist-test-slave-70st"
server_key: "a49c6d1afc4dead2b402bb63d1e0ae22"
server_key_iv: "e17488ba708b0bf06cb753265866c484"
server_key_version: "encryptionkey@0"
I20260924 04:58:51.336472 640 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 04:58:51.337075 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:58:51.337222 640 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 04:58:51.337298 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225929848825-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225929848825-392-0/minicluster-data/ts-1/wal
uuid: "0f34bf2209e94c938c7ceea1a335da2b"
format_stamp: "Formatted at 2026-09-24 04:58:51 on dist-test-slave-70st"
server_key: "a49c6d1afc4dead2b402bb63d1e0ae22"
server_key_iv: "e17488ba708b0bf06cb753265866c484"
server_key_version: "encryptionkey@0"
I20260924 04:58:51.337363 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225929848825-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225929848825-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225929848825-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:58:51.345639 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:58:51.347236 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:58:51.347358 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:58:51.347556 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:58:51.347826 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:58:51.347857 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:58:51.347884 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:58:51.347908 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:58:51.354058 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42831
I20260924 04:58:51.354094 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42831 every 8 connection(s)
I20260924 04:58:51.354393 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225929848825-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:58:51.358268 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 640
I20260924 04:58:51.358357 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225929848825-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:58:51.358494 392 external_mini_cluster.cc:1428] Setting key 8eb64730d667c0f89e289149fbca8408
Sep 24 04:58:51 dist-test-slave-70st krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225931, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:58:51.362818 502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:58:51.355019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60917 (local address 127.0.98.62:35327)
0924 04:58:51.355079 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 04:58:51.355082 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:58:51.355289 (+ 207us) server_negotiation.cc:407] Connection header received
0924 04:58:51.355744 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:58:51.355744 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:58:51.355755 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:58:51.355801 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:58:51.356654 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:58:51.357126 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:58:51.358061 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:58:51.358250 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:58:51.358304 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:58:51.360463 (+ 2159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:58:51.360465 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:58:51.360465 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:58:51.360472 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:58:51.361667 (+ 1195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:58:51.362001 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:58:51.362003 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:58:51.362003 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:58:51.362050 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:58:51.362256 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:58:51.362257 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:58:51.362257 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:58:51.362357 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:58:51.362410 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 04:58:51.362653 (+ 243us) server_negotiation.cc:299] Negotiation successful
0924 04:58:51.362731 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 04:58:51.362984 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:58:51.355031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35327 (local address 127.0.98.2:60917)
0924 04:58:51.355171 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 04:58:51.355205 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 04:58:51.355669 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:58:51.355865 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:58:51.355868 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:58:51.356396 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:58:51.356588 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:58:51.356593 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:58:51.357191 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:58:51.357191 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:58:51.358002 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:58:51.358004 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:58:51.358327 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:58:51.358328 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:58:51.358462 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:58:51.358923 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:58:51.358931 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:58:51.360371 (+ 1440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:58:51.361731 (+ 1360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:58:51.361737 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:58:51.361739 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:58:51.361932 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:58:51.362100 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:58:51.362101 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:58:51.362101 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:58:51.362204 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:58:51.362409 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:58:51.362415 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:58:51.362541 (+ 126us) client_negotiation.cc:769] Sending connection context
0924 04:58:51.362626 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 04:58:51.362773 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":81,"threads_started":1}
I20260924 04:58:51.363363 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35327
I20260924 04:58:51.363451 769 heartbeater.cc:464] Registering TS with master...
I20260924 04:58:51.363611 769 heartbeater.cc:511] Master 127.0.98.62:35327 requested a full tablet report, sending...
I20260924 04:58:51.363950 444 ts_manager.cc:194] Registered new tserver with Master: 0f34bf2209e94c938c7ceea1a335da2b (127.0.98.2:42831)
I20260924 04:58:51.364444 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60917
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225929848825-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225929848825-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:58:51.380275 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225929848825-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225929848825-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225929848825-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225929848825-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35327
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225929848825-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:58:51.458386 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:58:51.458580 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:58:51.458601 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:58:51.459669 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:58:51.459702 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:58:51.459724 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:58:51.461059 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225929848825-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225929848825-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225929848825-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225929848825-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35327
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225929848825-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 04:58:51.461239 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:58:51.461437 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:58:51.466014 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:58:51.466048 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:58:51.466060 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:58:51.466090 774 server_base.cc:1034] running on GCE node
I20260924 04:58:51.466451 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:58:51.466657 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:58:51.467804 774 hybrid_clock.cc:648] HybridClock initialized: now 1790225931467792 us; error 28 us; skew 500 ppm
Sep 24 04:58:51 dist-test-slave-70st krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225931, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:58:51.470062 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:58:51.470480 774 webserver.cc:460] Webserver started at http://127.0.98.3:37047/ using document root <none> and password file <none>
I20260924 04:58:51.470649 774 fs_manager.cc:359] Metadata directory not provided
I20260924 04:58:51.470685 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:58:51.470777 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:58:51.471797 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225929848825-392-0/minicluster-data/ts-2/data/instance:
uuid: "b9c68e476da3481ebbfcbef5f6a89484"
format_stamp: "Formatted at 2026-09-24 04:58:51 on dist-test-slave-70st"
server_key: "737769469841dfe53d99fd4d03fcf57e"
server_key_iv: "e465e626cccfdf60034d5890c4b7dc21"
server_key_version: "encryptionkey@0"
I20260924 04:58:51.472090 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225929848825-392-0/minicluster-data/ts-2/wal/instance:
uuid: "b9c68e476da3481ebbfcbef5f6a89484"
format_stamp: "Formatted at 2026-09-24 04:58:51 on dist-test-slave-70st"
server_key: "737769469841dfe53d99fd4d03fcf57e"
server_key_iv: "e465e626cccfdf60034d5890c4b7dc21"
server_key_version: "encryptionkey@0"
I20260924 04:58:51.473244 774 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 04:58:51.473997 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:58:51.474129 774 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 04:58:51.474207 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225929848825-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225929848825-392-0/minicluster-data/ts-2/wal
uuid: "b9c68e476da3481ebbfcbef5f6a89484"
format_stamp: "Formatted at 2026-09-24 04:58:51 on dist-test-slave-70st"
server_key: "737769469841dfe53d99fd4d03fcf57e"
server_key_iv: "e465e626cccfdf60034d5890c4b7dc21"
server_key_version: "encryptionkey@0"
I20260924 04:58:51.474291 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225929848825-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225929848825-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225929848825-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:58:51.498646 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:58:51.500540 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:58:51.500665 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:58:51.500873 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:58:51.501173 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:58:51.501206 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:58:51.501228 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:58:51.501252 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:58:51.507684 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34365
I20260924 04:58:51.507750 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34365 every 8 connection(s)
I20260924 04:58:51.508023 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225929848825-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 04:58:51 dist-test-slave-70st krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225931, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:58:51.514401 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 774
I20260924 04:58:51.514514 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790225929848825-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:58:51.514672 392 external_mini_cluster.cc:1428] Setting key 595d436cb26bf5cf17b3d76729d6df54
I20260924 04:58:51.516686 502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:58:51.508700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44141 (local address 127.0.98.62:35327)
0924 04:58:51.508762 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 04:58:51.508763 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 04:58:51.508946 (+ 183us) server_negotiation.cc:407] Connection header received
0924 04:58:51.509456 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:58:51.509458 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:58:51.509470 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:58:51.509526 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:58:51.510439 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:58:51.510933 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:58:51.511874 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:58:51.512071 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:58:51.512123 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:58:51.514064 (+ 1941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:58:51.514065 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:58:51.514065 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:58:51.514071 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:58:51.515457 (+ 1386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:58:51.515862 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:58:51.515863 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:58:51.515863 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:58:51.515918 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:58:51.516120 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:58:51.516121 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:58:51.516121 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:58:51.516227 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:58:51.516280 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 04:58:51.516530 (+ 250us) server_negotiation.cc:299] Negotiation successful
0924 04:58:51.516600 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 04:58:51.516849 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:58:51.508684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35327 (local address 127.0.98.3:44141)
0924 04:58:51.508829 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 04:58:51.508872 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 04:58:51.509347 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:58:51.509609 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:58:51.509612 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:58:51.510158 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:58:51.510352 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:58:51.510356 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:58:51.511008 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:58:51.511009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:58:51.511798 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:58:51.511800 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:58:51.512133 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:58:51.512134 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:58:51.512244 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:58:51.512662 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:58:51.512669 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:58:51.513985 (+ 1316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:58:51.515519 (+ 1534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:58:51.515524 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:58:51.515527 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:58:51.515796 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:58:51.515969 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:58:51.515969 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:58:51.515969 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:58:51.516068 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:58:51.516284 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:58:51.516290 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:58:51.516417 (+ 127us) client_negotiation.cc:769] Sending connection context
0924 04:58:51.516503 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 04:58:51.516648 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":68,"threads_started":1}
I20260924 04:58:51.517243 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35327
I20260924 04:58:51.517328 903 heartbeater.cc:464] Registering TS with master...
I20260924 04:58:51.517498 903 heartbeater.cc:511] Master 127.0.98.62:35327 requested a full tablet report, sending...
I20260924 04:58:51.517807 444 ts_manager.cc:194] Registered new tserver with Master: b9c68e476da3481ebbfcbef5f6a89484 (127.0.98.3:34365)
I20260924 04:58:51.518128 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:44141
I20260924 04:58:51.526978 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 04:58:51 dist-test-slave-70st krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225931, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 04:58:51 dist-test-slave-70st krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225931, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 04:58:51.548475 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:58:51.541203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35934 (local address 127.0.98.1:43453)
0924 04:58:51.541343 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 04:58:51.541347 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:58:51.541671 (+ 324us) server_negotiation.cc:407] Connection header received
0924 04:58:51.541725 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:58:51.541727 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:58:51.541741 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:58:51.541807 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:58:51.542314 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:58:51.543038 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:58:51.543709 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:58:51.544019 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:58:51.544067 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:58:51.546097 (+ 2030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:58:51.546105 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:58:51.546105 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:58:51.546111 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:58:51.547405 (+ 1294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:58:51.547697 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:58:51.547700 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:58:51.547700 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:58:51.547786 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:58:51.547960 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:58:51.547961 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:58:51.547962 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:58:51.548089 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:58:51.548213 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0924 04:58:51.548295 (+ 82us) server_negotiation.cc:299] Negotiation successful
0924 04:58:51.548368 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":58,"threads_started":1}
W20260924 04:58:51.548790 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35934
Sep 24 04:58:51 dist-test-slave-70st krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225931, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 04:58:51 dist-test-slave-70st krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225931, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 04:58:51.569483 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:58:51.563166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35946 (local address 127.0.98.1:43453)
0924 04:58:51.563225 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 04:58:51.563227 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:58:51.563236 (+ 9us) server_negotiation.cc:407] Connection header received
0924 04:58:51.563299 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:58:51.563300 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:58:51.563311 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:58:51.563359 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:58:51.563891 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:58:51.564417 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:58:51.565126 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:58:51.565329 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:58:51.565372 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:58:51.567436 (+ 2064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:58:51.567442 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:58:51.567442 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:58:51.567448 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:58:51.568693 (+ 1245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:58:51.568941 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:58:51.568942 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:58:51.568942 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:58:51.568988 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:58:51.569112 (+ 124us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:58:51.569112 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:58:51.569112 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:58:51.569187 (+ 75us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:58:51.569281 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 04:58:51.569311 (+ 30us) server_negotiation.cc:299] Negotiation successful
0924 04:58:51.569376 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 04:58:51.570001 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 506
I20260924 04:58:51.574700 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 640
I20260924 04:58:51.579385 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 774
I20260924 04:58:51.583137 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 412
2026-09-24T04:58:51Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1739 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225929848825-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:58:51 dist-test-slave-70st krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:58:51 dist-test-slave-70st krb5kdc[917](info): set up 2 sockets
Sep 24 04:58:51 dist-test-slave-70st krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260924 04:58:52.528859 392 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225929848825-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225929848825-392-0/krb5kdc/test-user.keytab.
Sep 24 04:58:52 dist-test-slave-70st krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225932, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:58:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:58:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:58:52.639985 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225929848825-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225929848825-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225929848825-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42021
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42021
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:58:52.718278 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:58:52.718485 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:58:52.718506 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:58:52.719606 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:58:52.719642 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:58:52.719655 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:58:52.719666 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:58:52.719676 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:58:52.721063 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225929848825-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225929848825-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42021
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42021
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225929848825-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 04:58:52.721247 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:58:52.721441 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:58:52.725848 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:58:52.725872 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:58:52.725978 930 server_base.cc:1034] running on GCE node
W20260924 04:58:52.725872 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:58:52.726233 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:58:52.726527 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:58:52.727679 930 hybrid_clock.cc:648] HybridClock initialized: now 1790225932727646 us; error 53 us; skew 500 ppm
Sep 24 04:58:52 dist-test-slave-70st krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225932, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:58:52.730041 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:58:52.730466 930 webserver.cc:460] Webserver started at http://127.0.98.62:43501/ using document root <none> and password file <none>
I20260924 04:58:52.730674 930 fs_manager.cc:359] Metadata directory not provided
I20260924 04:58:52.730720 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:58:52.730837 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:58:52.731849 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225929848825-392-0/minicluster-data/master-0/data/instance:
uuid: "50494b7d069a47c78be36879b2816540"
format_stamp: "Formatted at 2026-09-24 04:58:52 on dist-test-slave-70st"
server_key: "49815ed1b68f0e2055c72922a581f7e2"
server_key_iv: "debdd6a691043a79c8b298f9570f49b6"
server_key_version: "encryptionkey@0"
I20260924 04:58:52.732172 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225929848825-392-0/minicluster-data/master-0/wal/instance:
uuid: "50494b7d069a47c78be36879b2816540"
format_stamp: "Formatted at 2026-09-24 04:58:52 on dist-test-slave-70st"
server_key: "49815ed1b68f0e2055c72922a581f7e2"
server_key_iv: "debdd6a691043a79c8b298f9570f49b6"
server_key_version: "encryptionkey@0"
I20260924 04:58:52.733464 930 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 04:58:52.734141 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:58:52.734282 930 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 04:58:52.734360 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225929848825-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225929848825-392-0/minicluster-data/master-0/wal
uuid: "50494b7d069a47c78be36879b2816540"
format_stamp: "Formatted at 2026-09-24 04:58:52 on dist-test-slave-70st"
server_key: "49815ed1b68f0e2055c72922a581f7e2"
server_key_iv: "debdd6a691043a79c8b298f9570f49b6"
server_key_version: "encryptionkey@0"
I20260924 04:58:52.734422 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225929848825-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225929848825-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225929848825-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:58:52.758841 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:58:52.760677 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:58:52.760813 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:58:52.764531 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42021
I20260924 04:58:52.764573 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42021 every 8 connection(s)
I20260924 04:58:52.764899 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:58:52.765813 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:58:52.768362 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50494b7d069a47c78be36879b2816540: Bootstrap starting.
I20260924 04:58:52.769052 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 50494b7d069a47c78be36879b2816540: Neither blocks nor log segments found. Creating new log.
I20260924 04:58:52.769323 998 log.cc:824] T 00000000000000000000000000000000 P 50494b7d069a47c78be36879b2816540: Log is configured to *not* fsync() on all Append() calls
I20260924 04:58:52.770466 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50494b7d069a47c78be36879b2816540: No bootstrap required, opened a new log
I20260924 04:58:52.771989 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 50494b7d069a47c78be36879b2816540 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50494b7d069a47c78be36879b2816540" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42021 } }
I20260924 04:58:52.772125 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 50494b7d069a47c78be36879b2816540 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:58:52.772157 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 50494b7d069a47c78be36879b2816540 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50494b7d069a47c78be36879b2816540, State: Initialized, Role: FOLLOWER
I20260924 04:58:52.772305 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50494b7d069a47c78be36879b2816540 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50494b7d069a47c78be36879b2816540" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42021 } }
I20260924 04:58:52.772392 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 50494b7d069a47c78be36879b2816540 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:58:52.772429 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 50494b7d069a47c78be36879b2816540 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:58:52.772476 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50494b7d069a47c78be36879b2816540 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:58:52.772962 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 50494b7d069a47c78be36879b2816540 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50494b7d069a47c78be36879b2816540" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42021 } }
I20260924 04:58:52.773056 998 leader_election.cc:302] T 00000000000000000000000000000000 P 50494b7d069a47c78be36879b2816540 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 50494b7d069a47c78be36879b2816540; no voters:
I20260924 04:58:52.773211 998 leader_election.cc:288] T 00000000000000000000000000000000 P 50494b7d069a47c78be36879b2816540 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:58:52.773250 1001 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 50494b7d069a47c78be36879b2816540 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:58:52.773499 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 50494b7d069a47c78be36879b2816540 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:58:52.773649 1001 raft_consensus.cc:686] T 00000000000000000000000000000000 P 50494b7d069a47c78be36879b2816540 [term 1 LEADER]: Becoming Leader. State: Replica: 50494b7d069a47c78be36879b2816540, State: Running, Role: LEADER
I20260924 04:58:52.773739 1001 consensus_queue.cc:237] T 00000000000000000000000000000000 P 50494b7d069a47c78be36879b2816540 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50494b7d069a47c78be36879b2816540" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42021 } }
I20260924 04:58:52.774089 1002 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50494b7d069a47c78be36879b2816540 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "50494b7d069a47c78be36879b2816540" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50494b7d069a47c78be36879b2816540" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42021 } } }
I20260924 04:58:52.774098 1003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50494b7d069a47c78be36879b2816540 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 50494b7d069a47c78be36879b2816540. Latest consensus state: current_term: 1 leader_uuid: "50494b7d069a47c78be36879b2816540" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50494b7d069a47c78be36879b2816540" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42021 } } }
I20260924 04:58:52.774188 1002 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50494b7d069a47c78be36879b2816540 [sys.catalog]: This master's current role is: LEADER
I20260924 04:58:52.774194 1003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50494b7d069a47c78be36879b2816540 [sys.catalog]: This master's current role is: LEADER
I20260924 04:58:52.774577 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 930
I20260924 04:58:52.774665 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225929848825-392-0/minicluster-data/master-0/wal/instance
I20260924 04:58:52.774827 392 external_mini_cluster.cc:1428] Setting key 63ab74fb9ca5240a7fed03088fabddc8
I20260924 04:58:52.774829 1010 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:58:52.775715 1010 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:58:52.777065 1010 catalog_manager.cc:1347] Generated new cluster ID: 5856f3f1d56349e193be9caab7b7cc00
I20260924 04:58:52.777112 1010 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 04:58:52 dist-test-slave-70st krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225932, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:58:52.791844 1019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:58:52.775275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52010 (local address 127.0.98.62:42021)
0924 04:58:52.775407 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 04:58:52.775417 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 04:58:52.776315 (+ 898us) server_negotiation.cc:407] Connection header received
0924 04:58:52.776474 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:58:52.776481 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:58:52.776548 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:58:52.776694 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:58:52.781973 (+ 5279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:58:52.783440 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:58:52.784578 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:58:52.785274 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:58:52.785391 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:58:52.788206 (+ 2815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:58:52.788211 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:58:52.788211 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:58:52.788225 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:58:52.790245 (+ 2020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:58:52.790661 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:58:52.790665 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:58:52.790665 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:58:52.790774 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:58:52.790958 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:58:52.790958 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:58:52.790959 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:58:52.791247 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:58:52.791394 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 04:58:52.791482 (+ 88us) server_negotiation.cc:299] Negotiation successful
0924 04:58:52.791614 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":53,"threads_started":1}
I20260924 04:58:52.815186 1010 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:58:52.815728 1010 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:58:52.818116 1010 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 50494b7d069a47c78be36879b2816540: Generated new TSK 0
I20260924 04:58:52.818300 1010 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225929848825-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225929848825-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:58:52.847613 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225929848825-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225929848825-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225929848825-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225929848825-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42021
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225929848825-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:58:52.925794 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:58:52.925992 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:58:52.926014 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:58:52.927134 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:58:52.927169 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:58:52.927193 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:58:52.928578 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225929848825-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225929848825-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225929848825-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225929848825-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42021
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225929848825-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 04:58:52.928767 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:58:52.928934 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:58:52.933575 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:58:52.933598 1024 server_base.cc:1034] running on GCE node
W20260924 04:58:52.933701 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:58:52.933643 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:58:52.934193 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:58:52.934449 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:58:52.935599 1024 hybrid_clock.cc:648] HybridClock initialized: now 1790225932935584 us; error 31 us; skew 500 ppm
Sep 24 04:58:52 dist-test-slave-70st krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225932, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:58:52.938102 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:58:52.938530 1024 webserver.cc:460] Webserver started at http://127.0.98.1:44417/ using document root <none> and password file <none>
I20260924 04:58:52.938733 1024 fs_manager.cc:359] Metadata directory not provided
I20260924 04:58:52.938774 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:58:52.938898 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:58:52.939900 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225929848825-392-0/minicluster-data/ts-0/data/instance:
uuid: "ad1d7f78be92468a85971b55f2db54b7"
format_stamp: "Formatted at 2026-09-24 04:58:52 on dist-test-slave-70st"
server_key: "ce84dcfd9363fec0663fbca11034cc7b"
server_key_iv: "4a39ba3e6d77112f01c0d87a75cce9fc"
server_key_version: "encryptionkey@0"
I20260924 04:58:52.940222 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225929848825-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ad1d7f78be92468a85971b55f2db54b7"
format_stamp: "Formatted at 2026-09-24 04:58:52 on dist-test-slave-70st"
server_key: "ce84dcfd9363fec0663fbca11034cc7b"
server_key_iv: "4a39ba3e6d77112f01c0d87a75cce9fc"
server_key_version: "encryptionkey@0"
I20260924 04:58:52.941390 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 04:58:52.942143 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:58:52.942289 1024 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 04:58:52.942370 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225929848825-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225929848825-392-0/minicluster-data/ts-0/wal
uuid: "ad1d7f78be92468a85971b55f2db54b7"
format_stamp: "Formatted at 2026-09-24 04:58:52 on dist-test-slave-70st"
server_key: "ce84dcfd9363fec0663fbca11034cc7b"
server_key_iv: "4a39ba3e6d77112f01c0d87a75cce9fc"
server_key_version: "encryptionkey@0"
I20260924 04:58:52.942435 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225929848825-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225929848825-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225929848825-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:58:52.959609 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:58:52.961552 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:58:52.961692 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:58:52.961906 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:58:52.962205 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:58:52.962239 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:58:52.962266 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:58:52.962283 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:58:52.967746 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39355
I20260924 04:58:52.967792 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39355 every 8 connection(s)
I20260924 04:58:52.968111 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225929848825-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:58:52.971962 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 1024
I20260924 04:58:52.972043 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225929848825-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:58:52.972175 392 external_mini_cluster.cc:1428] Setting key e4aef6d7b949d4ea4c15968b3a1ee651
Sep 24 04:58:52 dist-test-slave-70st krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225932, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:58:52.977643 1019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:58:52.968799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44059 (local address 127.0.98.62:42021)
0924 04:58:52.968863 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 04:58:52.968866 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:58:52.969098 (+ 232us) server_negotiation.cc:407] Connection header received
0924 04:58:52.969624 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:58:52.969626 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:58:52.969637 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:58:52.969693 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:58:52.970632 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:58:52.971130 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:58:52.972139 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:58:52.972420 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:58:52.972489 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:58:52.974797 (+ 2308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:58:52.974798 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:58:52.974799 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:58:52.974807 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:58:52.976095 (+ 1288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:58:52.976545 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:58:52.976546 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:58:52.976546 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:58:52.976592 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:58:52.976885 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:58:52.976886 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:58:52.976886 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:58:52.977021 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:58:52.977094 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 04:58:52.977486 (+ 392us) server_negotiation.cc:299] Negotiation successful
0924 04:58:52.977574 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 04:58:52.977845 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:58:52.968772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42021 (local address 127.0.98.1:44059)
0924 04:58:52.968947 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 04:58:52.968987 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 04:58:52.969507 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:58:52.969757 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:58:52.969761 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:58:52.970321 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:58:52.970532 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:58:52.970537 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:58:52.971222 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:58:52.971223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:58:52.972058 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:58:52.972060 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:58:52.972496 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:58:52.972497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:58:52.972610 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:58:52.973161 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:58:52.973170 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:58:52.974690 (+ 1520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:58:52.976171 (+ 1481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:58:52.976177 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:58:52.976180 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:58:52.976474 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:58:52.976645 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:58:52.976646 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:58:52.976646 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:58:52.976784 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:58:52.977100 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:58:52.977106 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:58:52.977280 (+ 174us) client_negotiation.cc:769] Sending connection context
0924 04:58:52.977399 (+ 119us) client_negotiation.cc:240] Negotiation successful
0924 04:58:52.977599 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":96,"threads_started":1}
I20260924 04:58:52.978374 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42021
I20260924 04:58:52.978474 1153 heartbeater.cc:464] Registering TS with master...
I20260924 04:58:52.978667 1153 heartbeater.cc:511] Master 127.0.98.62:42021 requested a full tablet report, sending...
I20260924 04:58:52.979233 962 ts_manager.cc:194] Registered new tserver with Master: ad1d7f78be92468a85971b55f2db54b7 (127.0.98.1:39355)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 04:58:52.980142 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:44059
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225929848825-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225929848825-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:58:52.996510 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225929848825-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225929848825-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225929848825-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225929848825-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42021
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225929848825-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:58:53.078193 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:58:53.078389 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:58:53.078413 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:58:53.079489 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:58:53.079519 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:58:53.079545 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:58:53.080907 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225929848825-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225929848825-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225929848825-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225929848825-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42021
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225929848825-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 04:58:53.081086 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:58:53.081259 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:58:53.086047 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:58:53.086042 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:58:53.086042 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:58:53.086066 1158 server_base.cc:1034] running on GCE node
I20260924 04:58:53.086447 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:58:53.086685 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:58:53.087848 1158 hybrid_clock.cc:648] HybridClock initialized: now 1790225933087834 us; error 29 us; skew 500 ppm
Sep 24 04:58:53 dist-test-slave-70st krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225933, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:58:53.090369 1158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:58:53.090793 1158 webserver.cc:460] Webserver started at http://127.0.98.2:33677/ using document root <none> and password file <none>
I20260924 04:58:53.090974 1158 fs_manager.cc:359] Metadata directory not provided
I20260924 04:58:53.091017 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:58:53.091111 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:58:53.092119 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225929848825-392-0/minicluster-data/ts-1/data/instance:
uuid: "26f4686e73534c16afe37b04eb2cf221"
format_stamp: "Formatted at 2026-09-24 04:58:53 on dist-test-slave-70st"
server_key: "9c4e00efcc3b16092fc12003c75a04c7"
server_key_iv: "ca186550d5f028c7aaef7bab98e83991"
server_key_version: "encryptionkey@0"
I20260924 04:58:53.092450 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225929848825-392-0/minicluster-data/ts-1/wal/instance:
uuid: "26f4686e73534c16afe37b04eb2cf221"
format_stamp: "Formatted at 2026-09-24 04:58:53 on dist-test-slave-70st"
server_key: "9c4e00efcc3b16092fc12003c75a04c7"
server_key_iv: "ca186550d5f028c7aaef7bab98e83991"
server_key_version: "encryptionkey@0"
I20260924 04:58:53.093712 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 04:58:53.094393 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:58:53.094559 1158 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 04:58:53.094635 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225929848825-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225929848825-392-0/minicluster-data/ts-1/wal
uuid: "26f4686e73534c16afe37b04eb2cf221"
format_stamp: "Formatted at 2026-09-24 04:58:53 on dist-test-slave-70st"
server_key: "9c4e00efcc3b16092fc12003c75a04c7"
server_key_iv: "ca186550d5f028c7aaef7bab98e83991"
server_key_version: "encryptionkey@0"
I20260924 04:58:53.094687 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225929848825-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225929848825-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225929848825-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:58:53.106962 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:58:53.108814 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:58:53.108923 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:58:53.109097 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:58:53.109380 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:58:53.109450 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:58:53.109488 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:58:53.109506 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:58:53.115716 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37243
I20260924 04:58:53.115767 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37243 every 8 connection(s)
I20260924 04:58:53.116102 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225929848825-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:58:53.121129 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 1158
I20260924 04:58:53.121213 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225929848825-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:58:53.121376 392 external_mini_cluster.cc:1428] Setting key b6642ac5e6113c2305eb0a29ed702eed
Sep 24 04:58:53 dist-test-slave-70st krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225933, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:58:53.125473 1019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:58:53.116799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58881 (local address 127.0.98.62:42021)
0924 04:58:53.116853 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 04:58:53.116855 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:58:53.117046 (+ 191us) server_negotiation.cc:407] Connection header received
0924 04:58:53.117588 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:58:53.117590 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:58:53.117603 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:58:53.117655 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:58:53.118564 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:58:53.119055 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:58:53.120023 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:58:53.120211 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:58:53.120265 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:58:53.122576 (+ 2311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:58:53.122577 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:58:53.122577 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:58:53.122584 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:58:53.124109 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:58:53.124515 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:58:53.124516 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:58:53.124516 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:58:53.124567 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:58:53.124853 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:58:53.124853 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:58:53.124854 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:58:53.124950 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:58:53.124997 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 04:58:53.125304 (+ 307us) server_negotiation.cc:299] Negotiation successful
0924 04:58:53.125371 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 04:58:53.125735 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:58:53.116771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42021 (local address 127.0.98.2:58881)
0924 04:58:53.116911 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 04:58:53.116952 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 04:58:53.117489 (+ 537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:58:53.117726 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:58:53.117730 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:58:53.118277 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:58:53.118469 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:58:53.118474 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:58:53.119136 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:58:53.119137 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:58:53.119929 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:58:53.119930 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:58:53.120276 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:58:53.120277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:58:53.120403 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:58:53.120893 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:58:53.120901 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:58:53.122475 (+ 1574us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:58:53.124196 (+ 1721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:58:53.124202 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:58:53.124204 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:58:53.124434 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:58:53.124647 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:58:53.124647 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:58:53.124648 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:58:53.124787 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:58:53.125002 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:58:53.125007 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:58:53.125170 (+ 163us) client_negotiation.cc:769] Sending connection context
0924 04:58:53.125296 (+ 126us) client_negotiation.cc:240] Negotiation successful
0924 04:58:53.125505 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":73,"threads_started":1}
I20260924 04:58:53.126230 1287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42021
I20260924 04:58:53.126348 1287 heartbeater.cc:464] Registering TS with master...
I20260924 04:58:53.126546 1287 heartbeater.cc:511] Master 127.0.98.62:42021 requested a full tablet report, sending...
I20260924 04:58:53.126915 962 ts_manager.cc:194] Registered new tserver with Master: 26f4686e73534c16afe37b04eb2cf221 (127.0.98.2:37243)
I20260924 04:58:53.127281 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:58881
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225929848825-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225929848825-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:58:53.144919 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225929848825-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225929848825-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225929848825-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225929848825-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42021
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225929848825-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:58:53.224856 1292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:58:53.225075 1292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:58:53.225106 1292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:58:53.226277 1292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:58:53.226317 1292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:58:53.226351 1292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:58:53.227744 1292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225929848825-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225929848825-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225929848825-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225929848825-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42021
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225929848825-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 04:58:53.227977 1292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:58:53.228183 1292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:58:53.233047 1300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:58:53.233009 1298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:58:53.233093 1297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:58:53.233137 1292 server_base.cc:1034] running on GCE node
I20260924 04:58:53.233403 1292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:58:53.233654 1292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:58:53.234794 1292 hybrid_clock.cc:648] HybridClock initialized: now 1790225933234788 us; error 35 us; skew 500 ppm
Sep 24 04:58:53 dist-test-slave-70st krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225933, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:58:53.237109 1292 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:58:53.237596 1292 webserver.cc:460] Webserver started at http://127.0.98.3:44491/ using document root <none> and password file <none>
I20260924 04:58:53.237828 1292 fs_manager.cc:359] Metadata directory not provided
I20260924 04:58:53.237879 1292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:58:53.238009 1292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:58:53.239063 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225929848825-392-0/minicluster-data/ts-2/data/instance:
uuid: "fb87215c189d4b25a9be36bd2dffc666"
format_stamp: "Formatted at 2026-09-24 04:58:53 on dist-test-slave-70st"
server_key: "d90b9ccea58a5941e84db8d1f78ea5aa"
server_key_iv: "da4afaaf81fdb58acb0030abf0ff9b33"
server_key_version: "encryptionkey@0"
I20260924 04:58:53.239382 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225929848825-392-0/minicluster-data/ts-2/wal/instance:
uuid: "fb87215c189d4b25a9be36bd2dffc666"
format_stamp: "Formatted at 2026-09-24 04:58:53 on dist-test-slave-70st"
server_key: "d90b9ccea58a5941e84db8d1f78ea5aa"
server_key_iv: "da4afaaf81fdb58acb0030abf0ff9b33"
server_key_version: "encryptionkey@0"
I20260924 04:58:53.240664 1292 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 04:58:53.241291 1307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:58:53.241453 1292 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 04:58:53.241539 1292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225929848825-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225929848825-392-0/minicluster-data/ts-2/wal
uuid: "fb87215c189d4b25a9be36bd2dffc666"
format_stamp: "Formatted at 2026-09-24 04:58:53 on dist-test-slave-70st"
server_key: "d90b9ccea58a5941e84db8d1f78ea5aa"
server_key_iv: "da4afaaf81fdb58acb0030abf0ff9b33"
server_key_version: "encryptionkey@0"
I20260924 04:58:53.241604 1292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225929848825-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225929848825-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225929848825-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:58:53.252342 1292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:58:53.254221 1292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:58:53.254349 1292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:58:53.254545 1292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:58:53.254817 1292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:58:53.254849 1292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:58:53.254876 1292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:58:53.254894 1292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:58:53.260277 1292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43201
I20260924 04:58:53.260341 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43201 every 8 connection(s)
I20260924 04:58:53.260630 1292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225929848825-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 04:58:53 dist-test-slave-70st krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225933, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:58:53.269152 1019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:58:53.261287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51895 (local address 127.0.98.62:42021)
0924 04:58:53.261340 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 04:58:53.261342 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:58:53.261570 (+ 228us) server_negotiation.cc:407] Connection header received
0924 04:58:53.262080 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:58:53.262081 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:58:53.262092 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:58:53.262138 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:58:53.263064 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:58:53.263565 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:58:53.264530 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:58:53.264718 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:58:53.264777 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:58:53.266755 (+ 1978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:58:53.266758 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:58:53.266758 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:58:53.266773 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:58:53.267935 (+ 1162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:58:53.268303 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:58:53.268304 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:58:53.268305 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:58:53.268364 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:58:53.268576 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:58:53.268577 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:58:53.268577 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:58:53.268683 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:58:53.268739 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 04:58:53.268988 (+ 249us) server_negotiation.cc:299] Negotiation successful
0924 04:58:53.269063 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 04:58:53.269323 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:58:53.261259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42021 (local address 127.0.98.3:51895)
0924 04:58:53.261454 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0924 04:58:53.261500 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 04:58:53.261984 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:58:53.262209 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:58:53.262214 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:58:53.262774 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:58:53.262965 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:58:53.262970 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:58:53.263646 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:58:53.263647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:58:53.264447 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:58:53.264449 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:58:53.264780 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:58:53.264781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:58:53.264902 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:58:53.265367 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:58:53.265376 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:58:53.266672 (+ 1296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:58:53.268015 (+ 1343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:58:53.268022 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:58:53.268024 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:58:53.268237 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:58:53.268418 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:58:53.268418 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:58:53.268419 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:58:53.268521 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:58:53.268734 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:58:53.268740 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:58:53.268873 (+ 133us) client_negotiation.cc:769] Sending connection context
0924 04:58:53.268959 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 04:58:53.269107 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":86,"threads_started":1}
I20260924 04:58:53.269344 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 1292
I20260924 04:58:53.269454 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790225929848825-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:58:53.269637 392 external_mini_cluster.cc:1428] Setting key f321b6e48fa0736bc26792fbdda48f80
I20260924 04:58:53.269934 1421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42021
I20260924 04:58:53.270052 1421 heartbeater.cc:464] Registering TS with master...
I20260924 04:58:53.270277 1421 heartbeater.cc:511] Master 127.0.98.62:42021 requested a full tablet report, sending...
I20260924 04:58:53.270639 962 ts_manager.cc:194] Registered new tserver with Master: fb87215c189d4b25a9be36bd2dffc666 (127.0.98.3:43201)
I20260924 04:58:53.271005 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:51895
I20260924 04:58:53.281288 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:58:53.288130 1019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:58:53.282845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52026 (local address 127.0.98.62:42021)
0924 04:58:53.282913 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 04:58:53.282916 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:58:53.282977 (+ 61us) server_negotiation.cc:407] Connection header received
0924 04:58:53.283042 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:58:53.283043 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:58:53.283054 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:58:53.283105 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:58:53.283639 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:58:53.284119 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:58:53.284823 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:58:53.285005 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:58:53.285057 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:58:53.286133 (+ 1076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:58:53.286135 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:58:53.286136 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:58:53.286143 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:58:53.287144 (+ 1001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:58:53.287442 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:58:53.287443 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:58:53.287444 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:58:53.287495 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:58:53.287640 (+ 145us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:58:53.287641 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:58:53.287641 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:58:53.287779 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:58:53.287863 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 04:58:53.287972 (+ 109us) server_negotiation.cc:299] Negotiation successful
0924 04:58:53.288040 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 04:58:53.290192 962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52026:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:58:53.291740 962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:58:53.297806 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:58:53.293860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49628 (local address 127.0.98.3:43201)
0924 04:58:53.294010 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0924 04:58:53.294014 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:58:53.294119 (+ 105us) server_negotiation.cc:407] Connection header received
0924 04:58:53.294385 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:58:53.294387 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:58:53.294402 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:58:53.294470 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:58:53.294590 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:58:53.295370 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:58:53.296713 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:58:53.297369 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:58:53.297404 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:58:53.297484 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 04:58:53.297676 (+ 192us) server_negotiation.cc:299] Negotiation successful
0924 04:58:53.297716 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":56,"threads_started":1}
I20260924 04:58:53.297849 1429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:58:53.293715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43201 (local address 127.0.0.1:49628)
0924 04:58:53.294006 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0924 04:58:53.294049 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 04:58:53.294114 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:58:53.294467 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:58:53.294468 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:58:53.294470 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:58:53.294505 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:58:53.294506 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:58:53.295461 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:58:53.295461 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:58:53.296620 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:58:53.296623 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:58:53.297505 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:58:53.297505 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:58:53.297582 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:58:53.297591 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 04:58:53.297648 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 04:58:53.297713 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"mutex_wait_us":3,"spinlock_wait_cycles":384,"thread_start_us":54,"threads_started":1}
I20260924 04:58:53.297854 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:58:53.293715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37243 (local address 127.0.0.1:40214)
0924 04:58:53.294005 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0924 04:58:53.294048 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 04:58:53.294110 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:58:53.294536 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:58:53.294537 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:58:53.294539 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:58:53.294582 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:58:53.294584 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:58:53.295820 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:58:53.295821 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:58:53.296648 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:58:53.296650 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:58:53.297583 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:58:53.297585 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:58:53.297647 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:58:53.297651 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 04:58:53.297699 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 04:58:53.297740 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"mutex_wait_us":18,"thread_start_us":57,"threads_started":1}
I20260924 04:58:53.298060 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:58:53.293789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40214 (local address 127.0.98.2:37243)
0924 04:58:53.294219 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0924 04:58:53.294224 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:58:53.294232 (+ 8us) server_negotiation.cc:407] Connection header received
0924 04:58:53.294271 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:58:53.294273 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:58:53.294288 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:58:53.294364 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:58:53.294673 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:58:53.295747 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:58:53.296732 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:58:53.297410 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:58:53.297495 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:58:53.297550 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 04:58:53.297942 (+ 392us) server_negotiation.cc:299] Negotiation successful
0924 04:58:53.297994 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":95,"threads_started":1}
I20260924 04:58:53.298537 1430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:58:53.293904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39355 (local address 127.0.0.1:46834)
0924 04:58:53.294198 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0924 04:58:53.294206 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 04:58:53.294250 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:58:53.294420 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:58:53.294422 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:58:53.294424 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:58:53.294486 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:58:53.294488 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:58:53.295769 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:58:53.295770 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:58:53.296877 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:58:53.296879 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:58:53.298311 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:58:53.298312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:58:53.298380 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:58:53.298384 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 04:58:53.298441 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 04:58:53.298476 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"mutex_wait_us":38,"thread_start_us":53,"threads_started":1}
I20260924 04:58:53.298672 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:58:53.293998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46834 (local address 127.0.98.1:39355)
0924 04:58:53.294203 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0924 04:58:53.294207 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:58:53.294241 (+ 34us) server_negotiation.cc:407] Connection header received
0924 04:58:53.294322 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:58:53.294323 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:58:53.294342 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:58:53.294418 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:58:53.294544 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:58:53.295666 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:58:53.296957 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:58:53.297727 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:58:53.297781 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:58:53.297842 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 04:58:53.298470 (+ 628us) server_negotiation.cc:299] Negotiation successful
0924 04:58:53.298517 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":98,"threads_started":1}
I20260924 04:58:53.298606 1355 tablet_service.cc:1467] Processing CreateTablet for tablet e831eb7871d34ea4891614116e381b08 (DEFAULT_TABLE table=test-table [id=83d696cd5ec544ab81d4618fe79507d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:58:53.298713 1221 tablet_service.cc:1467] Processing CreateTablet for tablet e831eb7871d34ea4891614116e381b08 (DEFAULT_TABLE table=test-table [id=83d696cd5ec544ab81d4618fe79507d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:58:53.298959 1355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e831eb7871d34ea4891614116e381b08. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:58:53.299039 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e831eb7871d34ea4891614116e381b08. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:58:53.299222 1087 tablet_service.cc:1467] Processing CreateTablet for tablet e831eb7871d34ea4891614116e381b08 (DEFAULT_TABLE table=test-table [id=83d696cd5ec544ab81d4618fe79507d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:58:53.299499 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e831eb7871d34ea4891614116e381b08. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:58:53.301343 1435 tablet_bootstrap.cc:492] T e831eb7871d34ea4891614116e381b08 P fb87215c189d4b25a9be36bd2dffc666: Bootstrap starting.
I20260924 04:58:53.302031 1435 tablet_bootstrap.cc:654] T e831eb7871d34ea4891614116e381b08 P fb87215c189d4b25a9be36bd2dffc666: Neither blocks nor log segments found. Creating new log.
I20260924 04:58:53.302188 1436 tablet_bootstrap.cc:492] T e831eb7871d34ea4891614116e381b08 P 26f4686e73534c16afe37b04eb2cf221: Bootstrap starting.
I20260924 04:58:53.302577 1435 log.cc:824] T e831eb7871d34ea4891614116e381b08 P fb87215c189d4b25a9be36bd2dffc666: Log is configured to *not* fsync() on all Append() calls
I20260924 04:58:53.302631 1437 tablet_bootstrap.cc:492] T e831eb7871d34ea4891614116e381b08 P ad1d7f78be92468a85971b55f2db54b7: Bootstrap starting.
I20260924 04:58:53.302884 1436 tablet_bootstrap.cc:654] T e831eb7871d34ea4891614116e381b08 P 26f4686e73534c16afe37b04eb2cf221: Neither blocks nor log segments found. Creating new log.
I20260924 04:58:53.303193 1437 tablet_bootstrap.cc:654] T e831eb7871d34ea4891614116e381b08 P ad1d7f78be92468a85971b55f2db54b7: Neither blocks nor log segments found. Creating new log.
I20260924 04:58:53.303356 1435 tablet_bootstrap.cc:492] T e831eb7871d34ea4891614116e381b08 P fb87215c189d4b25a9be36bd2dffc666: No bootstrap required, opened a new log
I20260924 04:58:53.303426 1435 ts_tablet_manager.cc:1397] T e831eb7871d34ea4891614116e381b08 P fb87215c189d4b25a9be36bd2dffc666: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 04:58:53.303412 1436 log.cc:824] T e831eb7871d34ea4891614116e381b08 P 26f4686e73534c16afe37b04eb2cf221: Log is configured to *not* fsync() on all Append() calls
I20260924 04:58:53.303665 1437 log.cc:824] T e831eb7871d34ea4891614116e381b08 P ad1d7f78be92468a85971b55f2db54b7: Log is configured to *not* fsync() on all Append() calls
I20260924 04:58:53.304093 1436 tablet_bootstrap.cc:492] T e831eb7871d34ea4891614116e381b08 P 26f4686e73534c16afe37b04eb2cf221: No bootstrap required, opened a new log
I20260924 04:58:53.304162 1436 ts_tablet_manager.cc:1397] T e831eb7871d34ea4891614116e381b08 P 26f4686e73534c16afe37b04eb2cf221: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 04:58:53.304298 1437 tablet_bootstrap.cc:492] T e831eb7871d34ea4891614116e381b08 P ad1d7f78be92468a85971b55f2db54b7: No bootstrap required, opened a new log
I20260924 04:58:53.304353 1437 ts_tablet_manager.cc:1397] T e831eb7871d34ea4891614116e381b08 P ad1d7f78be92468a85971b55f2db54b7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 04:58:53.305244 1435 raft_consensus.cc:348] T e831eb7871d34ea4891614116e381b08 P fb87215c189d4b25a9be36bd2dffc666 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb87215c189d4b25a9be36bd2dffc666" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43201 } } peers { permanent_uuid: "ad1d7f78be92468a85971b55f2db54b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39355 } } peers { permanent_uuid: "26f4686e73534c16afe37b04eb2cf221" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37243 } }
I20260924 04:58:53.305436 1435 raft_consensus.cc:374] T e831eb7871d34ea4891614116e381b08 P fb87215c189d4b25a9be36bd2dffc666 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:58:53.305485 1435 raft_consensus.cc:729] T e831eb7871d34ea4891614116e381b08 P fb87215c189d4b25a9be36bd2dffc666 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb87215c189d4b25a9be36bd2dffc666, State: Initialized, Role: FOLLOWER
I20260924 04:58:53.305598 1435 consensus_queue.cc:260] T e831eb7871d34ea4891614116e381b08 P fb87215c189d4b25a9be36bd2dffc666 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb87215c189d4b25a9be36bd2dffc666" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43201 } } peers { permanent_uuid: "ad1d7f78be92468a85971b55f2db54b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39355 } } peers { permanent_uuid: "26f4686e73534c16afe37b04eb2cf221" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37243 } }
I20260924 04:58:53.305866 1435 ts_tablet_manager.cc:1428] T e831eb7871d34ea4891614116e381b08 P fb87215c189d4b25a9be36bd2dffc666: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 04:58:53.305989 1421 heartbeater.cc:503] Master 127.0.98.62:42021 was elected leader, sending a full tablet report...
I20260924 04:58:53.306115 1436 raft_consensus.cc:348] T e831eb7871d34ea4891614116e381b08 P 26f4686e73534c16afe37b04eb2cf221 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb87215c189d4b25a9be36bd2dffc666" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43201 } } peers { permanent_uuid: "ad1d7f78be92468a85971b55f2db54b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39355 } } peers { permanent_uuid: "26f4686e73534c16afe37b04eb2cf221" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37243 } }
I20260924 04:58:53.306250 1436 raft_consensus.cc:374] T e831eb7871d34ea4891614116e381b08 P 26f4686e73534c16afe37b04eb2cf221 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:58:53.306284 1436 raft_consensus.cc:729] T e831eb7871d34ea4891614116e381b08 P 26f4686e73534c16afe37b04eb2cf221 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26f4686e73534c16afe37b04eb2cf221, State: Initialized, Role: FOLLOWER
I20260924 04:58:53.305923 1437 raft_consensus.cc:348] T e831eb7871d34ea4891614116e381b08 P ad1d7f78be92468a85971b55f2db54b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb87215c189d4b25a9be36bd2dffc666" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43201 } } peers { permanent_uuid: "ad1d7f78be92468a85971b55f2db54b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39355 } } peers { permanent_uuid: "26f4686e73534c16afe37b04eb2cf221" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37243 } }
I20260924 04:58:53.306326 1437 raft_consensus.cc:374] T e831eb7871d34ea4891614116e381b08 P ad1d7f78be92468a85971b55f2db54b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:58:53.306375 1437 raft_consensus.cc:729] T e831eb7871d34ea4891614116e381b08 P ad1d7f78be92468a85971b55f2db54b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad1d7f78be92468a85971b55f2db54b7, State: Initialized, Role: FOLLOWER
I20260924 04:58:53.306396 1436 consensus_queue.cc:260] T e831eb7871d34ea4891614116e381b08 P 26f4686e73534c16afe37b04eb2cf221 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb87215c189d4b25a9be36bd2dffc666" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43201 } } peers { permanent_uuid: "ad1d7f78be92468a85971b55f2db54b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39355 } } peers { permanent_uuid: "26f4686e73534c16afe37b04eb2cf221" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37243 } }
I20260924 04:58:53.306492 1437 consensus_queue.cc:260] T e831eb7871d34ea4891614116e381b08 P ad1d7f78be92468a85971b55f2db54b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb87215c189d4b25a9be36bd2dffc666" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43201 } } peers { permanent_uuid: "ad1d7f78be92468a85971b55f2db54b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39355 } } peers { permanent_uuid: "26f4686e73534c16afe37b04eb2cf221" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37243 } }
I20260924 04:58:53.306632 1436 ts_tablet_manager.cc:1428] T e831eb7871d34ea4891614116e381b08 P 26f4686e73534c16afe37b04eb2cf221: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 04:58:53.306748 1287 heartbeater.cc:503] Master 127.0.98.62:42021 was elected leader, sending a full tablet report...
I20260924 04:58:53.307165 1437 ts_tablet_manager.cc:1428] T e831eb7871d34ea4891614116e381b08 P ad1d7f78be92468a85971b55f2db54b7: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260924 04:58:53.307188 1153 heartbeater.cc:503] Master 127.0.98.62:42021 was elected leader, sending a full tablet report...
I20260924 04:58:53.329403 1443 raft_consensus.cc:482] T e831eb7871d34ea4891614116e381b08 P ad1d7f78be92468a85971b55f2db54b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:58:53.329560 1443 raft_consensus.cc:504] T e831eb7871d34ea4891614116e381b08 P ad1d7f78be92468a85971b55f2db54b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb87215c189d4b25a9be36bd2dffc666" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43201 } } peers { permanent_uuid: "ad1d7f78be92468a85971b55f2db54b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39355 } } peers { permanent_uuid: "26f4686e73534c16afe37b04eb2cf221" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37243 } }
I20260924 04:58:53.329907 1443 leader_election.cc:288] T e831eb7871d34ea4891614116e381b08 P ad1d7f78be92468a85971b55f2db54b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fb87215c189d4b25a9be36bd2dffc666 (127.0.98.3:43201), 26f4686e73534c16afe37b04eb2cf221 (127.0.98.2:37243)
I20260924 04:58:53.333588 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:58:53.330126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55791 (local address 127.0.98.2:37243)
0924 04:58:53.330168 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 04:58:53.330170 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:58:53.330182 (+ 12us) server_negotiation.cc:407] Connection header received
0924 04:58:53.330227 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:58:53.330229 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:58:53.330238 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:58:53.330284 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:58:53.330431 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:58:53.331189 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:58:53.332645 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:58:53.333219 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:58:53.333285 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:58:53.333296 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 04:58:53.333479 (+ 183us) server_negotiation.cc:299] Negotiation successful
0924 04:58:53.333516 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 04:58:53.333588 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:58:53.330132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58375 (local address 127.0.98.3:43201)
0924 04:58:53.330179 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 04:58:53.330181 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:58:53.330290 (+ 109us) server_negotiation.cc:407] Connection header received
0924 04:58:53.330321 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:58:53.330322 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:58:53.330331 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:58:53.330362 (+ 31us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:58:53.330492 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:58:53.331182 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:58:53.332645 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:58:53.333226 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:58:53.333285 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:58:53.333296 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 04:58:53.333480 (+ 184us) server_negotiation.cc:299] Negotiation successful
0924 04:58:53.333516 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 04:58:53.333618 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:58:53.330040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37243 (local address 127.0.98.1:55791)
0924 04:58:53.330110 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0924 04:58:53.330122 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:58:53.330162 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:58:53.330295 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:58:53.330296 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:58:53.330298 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:58:53.330370 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:58:53.330372 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:58:53.331270 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:58:53.331271 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:58:53.332558 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:58:53.332562 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:58:53.333298 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:58:53.333299 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:58:53.333383 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:58:53.333387 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 04:58:53.333471 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 04:58:53.333529 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":44}
I20260924 04:58:53.333623 1444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:58:53.330046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43201 (local address 127.0.98.1:58375)
0924 04:58:53.330227 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0924 04:58:53.330235 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 04:58:53.330284 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:58:53.330365 (+ 81us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:58:53.330366 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:58:53.330367 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:58:53.330421 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:58:53.330422 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:58:53.331270 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:58:53.331271 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:58:53.332559 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:58:53.332562 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:58:53.333299 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:58:53.333299 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:58:53.333382 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:58:53.333389 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 04:58:53.333474 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 04:58:53.333527 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"mutex_wait_us":87,"thread_start_us":99,"threads_started":1}
I20260924 04:58:53.333972 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e831eb7871d34ea4891614116e381b08" candidate_uuid: "ad1d7f78be92468a85971b55f2db54b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26f4686e73534c16afe37b04eb2cf221" is_pre_election: true
I20260924 04:58:53.333999 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e831eb7871d34ea4891614116e381b08" candidate_uuid: "ad1d7f78be92468a85971b55f2db54b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb87215c189d4b25a9be36bd2dffc666" is_pre_election: true
I20260924 04:58:53.334126 1241 raft_consensus.cc:2454] T e831eb7871d34ea4891614116e381b08 P 26f4686e73534c16afe37b04eb2cf221 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad1d7f78be92468a85971b55f2db54b7 in term 0.
I20260924 04:58:53.334126 1375 raft_consensus.cc:2454] T e831eb7871d34ea4891614116e381b08 P fb87215c189d4b25a9be36bd2dffc666 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad1d7f78be92468a85971b55f2db54b7 in term 0.
I20260924 04:58:53.334311 1042 leader_election.cc:302] T e831eb7871d34ea4891614116e381b08 P ad1d7f78be92468a85971b55f2db54b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad1d7f78be92468a85971b55f2db54b7, fb87215c189d4b25a9be36bd2dffc666; no voters:
I20260924 04:58:53.334436 1443 raft_consensus.cc:2781] T e831eb7871d34ea4891614116e381b08 P ad1d7f78be92468a85971b55f2db54b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:58:53.334487 1443 raft_consensus.cc:482] T e831eb7871d34ea4891614116e381b08 P ad1d7f78be92468a85971b55f2db54b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:58:53.334513 1443 raft_consensus.cc:3037] T e831eb7871d34ea4891614116e381b08 P ad1d7f78be92468a85971b55f2db54b7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:58:53.335177 1443 raft_consensus.cc:504] T e831eb7871d34ea4891614116e381b08 P ad1d7f78be92468a85971b55f2db54b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb87215c189d4b25a9be36bd2dffc666" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43201 } } peers { permanent_uuid: "ad1d7f78be92468a85971b55f2db54b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39355 } } peers { permanent_uuid: "26f4686e73534c16afe37b04eb2cf221" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37243 } }
I20260924 04:58:53.335286 1443 leader_election.cc:288] T e831eb7871d34ea4891614116e381b08 P ad1d7f78be92468a85971b55f2db54b7 [CANDIDATE]: Term 1 election: Requested vote from peers fb87215c189d4b25a9be36bd2dffc666 (127.0.98.3:43201), 26f4686e73534c16afe37b04eb2cf221 (127.0.98.2:37243)
I20260924 04:58:53.335469 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e831eb7871d34ea4891614116e381b08" candidate_uuid: "ad1d7f78be92468a85971b55f2db54b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb87215c189d4b25a9be36bd2dffc666"
I20260924 04:58:53.335479 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e831eb7871d34ea4891614116e381b08" candidate_uuid: "ad1d7f78be92468a85971b55f2db54b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26f4686e73534c16afe37b04eb2cf221"
I20260924 04:58:53.335539 1241 raft_consensus.cc:3037] T e831eb7871d34ea4891614116e381b08 P 26f4686e73534c16afe37b04eb2cf221 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:58:53.335542 1375 raft_consensus.cc:3037] T e831eb7871d34ea4891614116e381b08 P fb87215c189d4b25a9be36bd2dffc666 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:58:53.336208 1241 raft_consensus.cc:2454] T e831eb7871d34ea4891614116e381b08 P 26f4686e73534c16afe37b04eb2cf221 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad1d7f78be92468a85971b55f2db54b7 in term 1.
I20260924 04:58:53.336215 1375 raft_consensus.cc:2454] T e831eb7871d34ea4891614116e381b08 P fb87215c189d4b25a9be36bd2dffc666 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad1d7f78be92468a85971b55f2db54b7 in term 1.
I20260924 04:58:53.336355 1042 leader_election.cc:302] T e831eb7871d34ea4891614116e381b08 P ad1d7f78be92468a85971b55f2db54b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad1d7f78be92468a85971b55f2db54b7, fb87215c189d4b25a9be36bd2dffc666; no voters:
I20260924 04:58:53.336438 1443 raft_consensus.cc:2781] T e831eb7871d34ea4891614116e381b08 P ad1d7f78be92468a85971b55f2db54b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:58:53.336580 1443 raft_consensus.cc:686] T e831eb7871d34ea4891614116e381b08 P ad1d7f78be92468a85971b55f2db54b7 [term 1 LEADER]: Becoming Leader. State: Replica: ad1d7f78be92468a85971b55f2db54b7, State: Running, Role: LEADER
I20260924 04:58:53.336669 1443 consensus_queue.cc:237] T e831eb7871d34ea4891614116e381b08 P ad1d7f78be92468a85971b55f2db54b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb87215c189d4b25a9be36bd2dffc666" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43201 } } peers { permanent_uuid: "ad1d7f78be92468a85971b55f2db54b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39355 } } peers { permanent_uuid: "26f4686e73534c16afe37b04eb2cf221" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37243 } }
I20260924 04:58:53.337549 962 catalog_manager.cc:5495] T e831eb7871d34ea4891614116e381b08 P ad1d7f78be92468a85971b55f2db54b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to ad1d7f78be92468a85971b55f2db54b7 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "ad1d7f78be92468a85971b55f2db54b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb87215c189d4b25a9be36bd2dffc666" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43201 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad1d7f78be92468a85971b55f2db54b7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26f4686e73534c16afe37b04eb2cf221" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37243 } health_report { overall_health: UNKNOWN } } }
Sep 24 04:58:53 dist-test-slave-70st krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225932, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 04:58:53.345822 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:58:53.339191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46842 (local address 127.0.98.1:39355)
0924 04:58:53.339224 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 04:58:53.339226 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:58:53.339341 (+ 115us) server_negotiation.cc:407] Connection header received
0924 04:58:53.339402 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:58:53.339403 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:58:53.339412 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:58:53.339448 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:58:53.339915 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:58:53.340342 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:58:53.340981 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:58:53.341149 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:58:53.341188 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:58:53.343246 (+ 2058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:58:53.343253 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:58:53.343254 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:58:53.343260 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:58:53.344793 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:58:53.345148 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:58:53.345151 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:58:53.345151 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:58:53.345208 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:58:53.345345 (+ 137us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:58:53.345346 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:58:53.345346 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:58:53.345517 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:58:53.345620 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 04:58:53.345699 (+ 79us) server_negotiation.cc:299] Negotiation successful
0924 04:58:53.345760 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
Sep 24 04:58:53 dist-test-slave-70st krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225933, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 04:58:53 dist-test-slave-70st krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225933, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 04:58:53.366122 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:58:53.359891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46848 (local address 127.0.98.1:39355)
0924 04:58:53.359950 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 04:58:53.359952 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:58:53.360015 (+ 63us) server_negotiation.cc:407] Connection header received
0924 04:58:53.360066 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:58:53.360067 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:58:53.360075 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:58:53.360121 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:58:53.360603 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:58:53.361096 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:58:53.361828 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:58:53.362031 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:58:53.362082 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:58:53.364172 (+ 2090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:58:53.364173 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:58:53.364174 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:58:53.364180 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:58:53.365221 (+ 1041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:58:53.365558 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:58:53.365559 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:58:53.365559 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:58:53.365608 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:58:53.365737 (+ 129us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:58:53.365737 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:58:53.365737 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:58:53.365824 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:58:53.365889 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 04:58:53.366003 (+ 114us) server_negotiation.cc:299] Negotiation successful
0924 04:58:53.366057 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
W20260924 04:58:53.366614 1288 tablet.cc:2369] T e831eb7871d34ea4891614116e381b08 P 26f4686e73534c16afe37b04eb2cf221: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 04:58:53 dist-test-slave-70st krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225933, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 04:58:53 dist-test-slave-70st krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225933, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 04:58:53.388446 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:58:53.381348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46864 (local address 127.0.98.1:39355)
0924 04:58:53.381455 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0924 04:58:53.381457 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:58:53.381491 (+ 34us) server_negotiation.cc:407] Connection header received
0924 04:58:53.381586 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:58:53.381587 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:58:53.381601 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:58:53.381657 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:58:53.382244 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:58:53.383000 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:58:53.383836 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:58:53.384027 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:58:53.384080 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:58:53.386246 (+ 2166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:58:53.386247 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:58:53.386248 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:58:53.386263 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:58:53.387516 (+ 1253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:58:53.387806 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:58:53.387806 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:58:53.387807 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:58:53.387857 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:58:53.388009 (+ 152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:58:53.388009 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:58:53.388009 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:58:53.388101 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:58:53.388187 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 04:58:53.388288 (+ 101us) server_negotiation.cc:299] Negotiation successful
0924 04:58:53.388356 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260924 04:58:53.388933 1067 tablet_service.cc:2908] Rejecting scan request for tablet e831eb7871d34ea4891614116e381b08: Uninitialized: clean time has not yet been initialized
I20260924 04:58:53.389671 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 1024
I20260924 04:58:53.394364 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 1158
I20260924 04:58:53.397835 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 1292
I20260924 04:58:53.401160 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 930
2026-09-24T04:58:53Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (1818 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225929848825-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:58:53 dist-test-slave-70st krb5kdc[1459](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:58:53 dist-test-slave-70st krb5kdc[1459](info): set up 2 sockets
Sep 24 04:58:53 dist-test-slave-70st krb5kdc[1459](info): commencing operation
krb5kdc: starting...
W20260924 04:58:54.455149 392 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260924 04:58:54.642757 392 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.188s user 0.000s sys 0.002s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260924 04:58:54.935091 392 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.292s user 0.000s sys 0.001s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260924 04:58:55.161967 392 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.227s user 0.000s sys 0.001s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225929848825-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225929848825-392-0/krb5kdc/test-user.keytab.
W20260924 04:58:56.099754 392 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.938s user 0.000s sys 0.001s
Sep 24 04:58:56 dist-test-slave-70st krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225936, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:58:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:58:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab.
W20260924 04:58:56.712348 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.62: real 0.593s user 0.000s sys 0.003s
I20260924 04:58:56.712672 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225929848825-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225929848825-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225929848825-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42079
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42079
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:58:56.793673 1472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:58:56.793941 1472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:58:56.793973 1472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:58:56.795639 1472 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:58:56.795691 1472 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 04:58:56.795710 1472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:58:56.795728 1472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:58:56.795753 1472 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:58:56.795769 1472 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:58:56.797885 1472 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225929848825-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225929848825-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42079
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42079
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1472
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225929848825-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 04:58:56.798099 1472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:58:56.798286 1472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:58:56.802850 1478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:58:56.802871 1480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:58:56.802932 1477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:58:56.802917 1472 server_base.cc:1034] running on GCE node
I20260924 04:58:56.803375 1472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:58:56.803668 1472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:58:56.804816 1472 hybrid_clock.cc:648] HybridClock initialized: now 1790225936804806 us; error 38 us; skew 500 ppm
Sep 24 04:58:56 dist-test-slave-70st krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225936, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:58:56.814858 1472 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:58:56.815312 1472 webserver.cc:460] Webserver started at http://127.0.98.62:36177/ using document root <none> and password file <none>
I20260924 04:58:56.815521 1472 fs_manager.cc:359] Metadata directory not provided
I20260924 04:58:56.815567 1472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:58:56.815690 1472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:58:56.816733 1472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225929848825-392-0/minicluster-data/master-0/data/instance:
uuid: "d11579be4182427e90eaae6d4647dff1"
format_stamp: "Formatted at 2026-09-24 04:58:56 on dist-test-slave-70st"
server_key: "9896d67e513b62ae07058a6c4a6b779e"
server_key_iv: "e5dee6eccd1e910dc2c5ec219446db30"
server_key_version: "encryptionkey@0"
I20260924 04:58:56.817047 1472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225929848825-392-0/minicluster-data/master-0/wal/instance:
uuid: "d11579be4182427e90eaae6d4647dff1"
format_stamp: "Formatted at 2026-09-24 04:58:56 on dist-test-slave-70st"
server_key: "9896d67e513b62ae07058a6c4a6b779e"
server_key_iv: "e5dee6eccd1e910dc2c5ec219446db30"
server_key_version: "encryptionkey@0"
I20260924 04:58:56.818377 1472 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 04:58:56.819074 1487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:58:56.819232 1472 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 04:58:56.819336 1472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225929848825-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225929848825-392-0/minicluster-data/master-0/wal
uuid: "d11579be4182427e90eaae6d4647dff1"
format_stamp: "Formatted at 2026-09-24 04:58:56 on dist-test-slave-70st"
server_key: "9896d67e513b62ae07058a6c4a6b779e"
server_key_iv: "e5dee6eccd1e910dc2c5ec219446db30"
server_key_version: "encryptionkey@0"
I20260924 04:58:56.819401 1472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225929848825-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225929848825-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225929848825-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:58:56.833232 1472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:58:56.835196 1472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:58:56.835335 1472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:58:56.839651 1472 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42079
I20260924 04:58:56.839696 1549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42079 every 8 connection(s)
I20260924 04:58:56.839998 1472 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:58:56.840616 1550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:58:56.842816 1550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d11579be4182427e90eaae6d4647dff1: Bootstrap starting.
I20260924 04:58:56.843357 1550 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d11579be4182427e90eaae6d4647dff1: Neither blocks nor log segments found. Creating new log.
I20260924 04:58:56.843619 1550 log.cc:824] T 00000000000000000000000000000000 P d11579be4182427e90eaae6d4647dff1: Log is configured to *not* fsync() on all Append() calls
I20260924 04:58:56.844234 1550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d11579be4182427e90eaae6d4647dff1: No bootstrap required, opened a new log
I20260924 04:58:56.846242 1550 raft_consensus.cc:348] T 00000000000000000000000000000000 P d11579be4182427e90eaae6d4647dff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d11579be4182427e90eaae6d4647dff1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42079 } }
I20260924 04:58:56.846382 1550 raft_consensus.cc:374] T 00000000000000000000000000000000 P d11579be4182427e90eaae6d4647dff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:58:56.846411 1550 raft_consensus.cc:729] T 00000000000000000000000000000000 P d11579be4182427e90eaae6d4647dff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d11579be4182427e90eaae6d4647dff1, State: Initialized, Role: FOLLOWER
I20260924 04:58:56.846532 1550 consensus_queue.cc:260] T 00000000000000000000000000000000 P d11579be4182427e90eaae6d4647dff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d11579be4182427e90eaae6d4647dff1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42079 } }
I20260924 04:58:56.846602 1550 raft_consensus.cc:388] T 00000000000000000000000000000000 P d11579be4182427e90eaae6d4647dff1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:58:56.846625 1550 raft_consensus.cc:482] T 00000000000000000000000000000000 P d11579be4182427e90eaae6d4647dff1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:58:56.846655 1550 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d11579be4182427e90eaae6d4647dff1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:58:56.847250 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 1472
I20260924 04:58:56.847321 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225929848825-392-0/minicluster-data/master-0/wal/instance
I20260924 04:58:56.847448 392 external_mini_cluster.cc:1428] Setting key b2bcfc547b1148842d2fa04660415db4
Sep 24 04:58:56 dist-test-slave-70st krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225936, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:58:56.856235 1554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:58:56.847936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42062 (local address 127.0.98.62:42079)
0924 04:58:56.848099 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 04:58:56.848107 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 04:58:56.848133 (+ 26us) server_negotiation.cc:407] Connection header received
0924 04:58:56.848232 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:58:56.848239 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:58:56.848302 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:58:56.848432 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:58:56.848945 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:58:56.850048 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:58:56.850781 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:58:56.851210 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:58:56.851302 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:58:56.853473 (+ 2171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:58:56.853477 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:58:56.853477 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:58:56.853488 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:58:56.854887 (+ 1399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:58:56.855207 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:58:56.855211 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:58:56.855211 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:58:56.855294 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:58:56.855459 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:58:56.855460 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:58:56.855460 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:58:56.855701 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:58:56.855837 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 04:58:56.855937 (+ 100us) server_negotiation.cc:299] Negotiation successful
0924 04:58:56.856065 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":85,"threads_started":1}
I20260924 04:58:56.865985 1550 raft_consensus.cc:504] T 00000000000000000000000000000000 P d11579be4182427e90eaae6d4647dff1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d11579be4182427e90eaae6d4647dff1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42079 } }
I20260924 04:58:56.866129 1550 leader_election.cc:302] T 00000000000000000000000000000000 P d11579be4182427e90eaae6d4647dff1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d11579be4182427e90eaae6d4647dff1; no voters:
I20260924 04:58:56.866310 1550 leader_election.cc:288] T 00000000000000000000000000000000 P d11579be4182427e90eaae6d4647dff1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:58:56.866379 1555 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d11579be4182427e90eaae6d4647dff1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:58:56.866580 1550 sys_catalog.cc:564] T 00000000000000000000000000000000 P d11579be4182427e90eaae6d4647dff1 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:58:56.866611 1555 raft_consensus.cc:686] T 00000000000000000000000000000000 P d11579be4182427e90eaae6d4647dff1 [term 1 LEADER]: Becoming Leader. State: Replica: d11579be4182427e90eaae6d4647dff1, State: Running, Role: LEADER
I20260924 04:58:56.866787 1555 consensus_queue.cc:237] T 00000000000000000000000000000000 P d11579be4182427e90eaae6d4647dff1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d11579be4182427e90eaae6d4647dff1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42079 } }
I20260924 04:58:56.867208 1556 sys_catalog.cc:455] T 00000000000000000000000000000000 P d11579be4182427e90eaae6d4647dff1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d11579be4182427e90eaae6d4647dff1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d11579be4182427e90eaae6d4647dff1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42079 } } }
I20260924 04:58:56.867331 1556 sys_catalog.cc:458] T 00000000000000000000000000000000 P d11579be4182427e90eaae6d4647dff1 [sys.catalog]: This master's current role is: LEADER
I20260924 04:58:56.867544 1557 sys_catalog.cc:455] T 00000000000000000000000000000000 P d11579be4182427e90eaae6d4647dff1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d11579be4182427e90eaae6d4647dff1. Latest consensus state: current_term: 1 leader_uuid: "d11579be4182427e90eaae6d4647dff1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d11579be4182427e90eaae6d4647dff1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42079 } } }
I20260924 04:58:56.867676 1557 sys_catalog.cc:458] T 00000000000000000000000000000000 P d11579be4182427e90eaae6d4647dff1 [sys.catalog]: This master's current role is: LEADER
I20260924 04:58:56.867837 1564 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:58:56.868404 1564 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:58:56.869818 1564 catalog_manager.cc:1347] Generated new cluster ID: 7b683efc8ecb472888a6df78c7b1d588
I20260924 04:58:56.869863 1564 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:58:56.881089 1564 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:58:56.881840 1564 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:58:56.895202 1564 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d11579be4182427e90eaae6d4647dff1: Generated new TSK 0
I20260924 04:58:56.895388 1564 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225929848825-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225929848825-392-0/krb5kdc/kudu_127.0.98.1.keytab.
W20260924 04:58:57.614923 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.1: real 0.710s user 0.000s sys 0.003s
I20260924 04:58:57.615237 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225929848825-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225929848825-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225929848825-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225929848825-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42079
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225929848825-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:58:57.695214 1576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:58:57.695406 1576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:58:57.695427 1576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:58:57.696561 1576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:58:57.696601 1576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:58:57.696620 1576 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 04:58:57.696636 1576 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:58:57.698222 1576 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225929848825-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225929848825-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225929848825-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225929848825-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:42079
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1576
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225929848825-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 04:58:57.698398 1576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:58:57.698566 1576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:58:57.703528 1581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:58:57.703550 1584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:58:57.703590 1582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:58:57.704005 1576 server_base.cc:1034] running on GCE node
I20260924 04:58:57.704195 1576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:58:57.704403 1576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:58:57.705528 1576 hybrid_clock.cc:648] HybridClock initialized: now 1790225937705515 us; error 29 us; skew 500 ppm
Sep 24 04:58:57 dist-test-slave-70st krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225937, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:58:57.707981 1576 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:58:57.708379 1576 webserver.cc:460] Webserver started at http://127.0.98.1:33895/ using document root <none> and password file <none>
I20260924 04:58:57.708568 1576 fs_manager.cc:359] Metadata directory not provided
I20260924 04:58:57.708614 1576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:58:57.708729 1576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:58:57.709849 1576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225929848825-392-0/minicluster-data/ts-0/data/instance:
uuid: "239a221e14974ed1be41fa89fe85928f"
format_stamp: "Formatted at 2026-09-24 04:58:57 on dist-test-slave-70st"
server_key: "d45b9f894a0ff303209b52a6ec6e5234"
server_key_iv: "f5a61905d6aa8692e9f0f2f577ec2028"
server_key_version: "encryptionkey@0"
I20260924 04:58:57.710165 1576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225929848825-392-0/minicluster-data/ts-0/wal/instance:
uuid: "239a221e14974ed1be41fa89fe85928f"
format_stamp: "Formatted at 2026-09-24 04:58:57 on dist-test-slave-70st"
server_key: "d45b9f894a0ff303209b52a6ec6e5234"
server_key_iv: "f5a61905d6aa8692e9f0f2f577ec2028"
server_key_version: "encryptionkey@0"
I20260924 04:58:57.711321 1576 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 04:58:57.711937 1591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:58:57.712060 1576 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 04:58:57.712138 1576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225929848825-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225929848825-392-0/minicluster-data/ts-0/wal
uuid: "239a221e14974ed1be41fa89fe85928f"
format_stamp: "Formatted at 2026-09-24 04:58:57 on dist-test-slave-70st"
server_key: "d45b9f894a0ff303209b52a6ec6e5234"
server_key_iv: "f5a61905d6aa8692e9f0f2f577ec2028"
server_key_version: "encryptionkey@0"
I20260924 04:58:57.712193 1576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225929848825-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225929848825-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225929848825-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:58:57.733984 1576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:58:57.735741 1576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:58:57.735903 1576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:58:57.736392 1576 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:58:57.736433 1576 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:58:57.736469 1576 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:58:57.736492 1576 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:58:57.743078 1576 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44377
Sep 24 04:58:57 dist-test-slave-70st krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225937, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:58:57.743160 1707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44377 every 8 connection(s)
I20260924 04:58:57.743465 1576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225929848825-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:58:57.746079 1600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:58:57.736947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43049 (local address 127.0.98.62:42079)
0924 04:58:57.737059 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0924 04:58:57.737062 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:58:57.737207 (+ 145us) server_negotiation.cc:407] Connection header received
0924 04:58:57.737799 (+ 592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:58:57.737800 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:58:57.737815 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:58:57.737875 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:58:57.738544 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:58:57.739303 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:58:57.740815 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:58:57.741073 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:58:57.741140 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:58:57.743598 (+ 2458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:58:57.743600 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:58:57.743600 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:58:57.743608 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:58:57.744959 (+ 1351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:58:57.745309 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:58:57.745309 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:58:57.745310 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:58:57.745366 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:58:57.745586 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:58:57.745587 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:58:57.745587 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:58:57.745689 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:58:57.745738 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 04:58:57.745941 (+ 203us) server_negotiation.cc:299] Negotiation successful
0924 04:58:57.746011 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":55,"threads_started":1}
I20260924 04:58:57.746200 1601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:58:57.736920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42079 (local address 127.0.98.1:43049)
0924 04:58:57.737090 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0924 04:58:57.737128 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 04:58:57.737699 (+ 571us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:58:57.737945 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:58:57.737950 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:58:57.738255 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:58:57.738448 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:58:57.738453 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:58:57.739396 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:58:57.739397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:58:57.740704 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:58:57.740707 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:58:57.741177 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:58:57.741178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:58:57.741319 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:58:57.741817 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:58:57.741828 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:58:57.743502 (+ 1674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:58:57.745045 (+ 1543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:58:57.745050 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:58:57.745052 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:58:57.745251 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:58:57.745404 (+ 153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:58:57.745405 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:58:57.745405 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:58:57.745539 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:58:57.745735 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:58:57.745737 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:58:57.745844 (+ 107us) client_negotiation.cc:769] Sending connection context
0924 04:58:57.745915 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 04:58:57.746029 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":79,"threads_started":1}
I20260924 04:58:57.746639 1708 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42079
I20260924 04:58:57.746714 1708 heartbeater.cc:464] Registering TS with master...
I20260924 04:58:57.746866 1708 heartbeater.cc:511] Master 127.0.98.62:42079 requested a full tablet report, sending...
I20260924 04:58:57.747388 1503 ts_manager.cc:194] Registered new tserver with Master: 239a221e14974ed1be41fa89fe85928f (127.0.98.1:44377)
I20260924 04:58:57.748257 1503 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:43049
I20260924 04:58:57.750222 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 1576
I20260924 04:58:57.750293 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225929848825-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:58:57.750483 392 external_mini_cluster.cc:1428] Setting key fe71b5a36025d9290ab1788cc644781e
I20260924 04:58:57.752532 1600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:58:57.747552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58471 (local address 127.0.98.62:42079)
0924 04:58:57.747588 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 04:58:57.747590 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:58:57.747667 (+ 77us) server_negotiation.cc:407] Connection header received
0924 04:58:57.747729 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:58:57.747729 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:58:57.747737 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:58:57.747780 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:58:57.748071 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:58:57.748589 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:58:57.749469 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:58:57.749660 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:58:57.749707 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:58:57.750427 (+ 720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:58:57.750428 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:58:57.750428 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:58:57.750434 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:58:57.751621 (+ 1187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:58:57.751900 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:58:57.751901 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:58:57.751901 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:58:57.751948 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:58:57.752098 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:58:57.752099 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:58:57.752099 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:58:57.752193 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:58:57.752243 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 04:58:57.752367 (+ 124us) server_negotiation.cc:299] Negotiation successful
0924 04:58:57.752437 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260924 04:58:57.752537 1714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:58:57.747490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42079 (local address 127.0.98.1:58471)
0924 04:58:57.747616 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 04:58:57.747625 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 04:58:57.747679 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:58:57.747784 (+ 105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:58:57.747785 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:58:57.747959 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:58:57.748019 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:58:57.748020 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:58:57.748683 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:58:57.748684 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:58:57.749355 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:58:57.749357 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:58:57.749711 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:58:57.749712 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:58:57.749755 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:58:57.750067 (+ 312us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:58:57.750071 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:58:57.750335 (+ 264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:58:57.751693 (+ 1358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:58:57.751694 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:58:57.751694 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:58:57.751841 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:58:57.751998 (+ 157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:58:57.751999 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:58:57.751999 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:58:57.752051 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:58:57.752242 (+ 191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:58:57.752243 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:58:57.752282 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 04:58:57.752349 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 04:58:57.752438 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":67,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225929848825-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225929848825-392-0/krb5kdc/kudu_127.0.98.2.keytab.
W20260924 04:58:58.606139 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.2: real 0.855s user 0.000s sys 0.003s
I20260924 04:58:58.606459 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225929848825-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225929848825-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225929848825-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225929848825-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42079
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225929848825-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:58:58.683429 1717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:58:58.683621 1717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:58:58.683650 1717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:58:58.684835 1717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:58:58.684875 1717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:58:58.684901 1717 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 04:58:58.684924 1717 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:58:58.686403 1717 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225929848825-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225929848825-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225929848825-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225929848825-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:42079
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1717
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225929848825-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 04:58:58.686625 1717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:58:58.686823 1717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:58:58.691386 1723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:58:58.691442 1722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:58:58.691464 1725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:58:58.691715 1717 server_base.cc:1034] running on GCE node
I20260924 04:58:58.691912 1717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:58:58.692168 1717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:58:58.693348 1717 hybrid_clock.cc:648] HybridClock initialized: now 1790225938693331 us; error 26 us; skew 500 ppm
Sep 24 04:58:58 dist-test-slave-70st krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225938, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:58:58.696146 1717 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:58:58.696823 1717 webserver.cc:460] Webserver started at http://127.0.98.2:36661/ using document root <none> and password file <none>
I20260924 04:58:58.697074 1717 fs_manager.cc:359] Metadata directory not provided
I20260924 04:58:58.697139 1717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:58:58.697268 1717 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:58:58.698638 1717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225929848825-392-0/minicluster-data/ts-1/data/instance:
uuid: "3856444a605042b69a7e3159ec5e35b0"
format_stamp: "Formatted at 2026-09-24 04:58:58 on dist-test-slave-70st"
server_key: "3316c2999e7167a795d080771ef547cc"
server_key_iv: "529b21e3b33c0f3a1fa4557e63ac23b7"
server_key_version: "encryptionkey@0"
I20260924 04:58:58.698961 1717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225929848825-392-0/minicluster-data/ts-1/wal/instance:
uuid: "3856444a605042b69a7e3159ec5e35b0"
format_stamp: "Formatted at 2026-09-24 04:58:58 on dist-test-slave-70st"
server_key: "3316c2999e7167a795d080771ef547cc"
server_key_iv: "529b21e3b33c0f3a1fa4557e63ac23b7"
server_key_version: "encryptionkey@0"
I20260924 04:58:58.700397 1717 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 04:58:58.701120 1732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:58:58.701272 1717 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 04:58:58.701351 1717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225929848825-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225929848825-392-0/minicluster-data/ts-1/wal
uuid: "3856444a605042b69a7e3159ec5e35b0"
format_stamp: "Formatted at 2026-09-24 04:58:58 on dist-test-slave-70st"
server_key: "3316c2999e7167a795d080771ef547cc"
server_key_iv: "529b21e3b33c0f3a1fa4557e63ac23b7"
server_key_version: "encryptionkey@0"
I20260924 04:58:58.701439 1717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225929848825-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225929848825-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225929848825-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:58:58.717017 1717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:58:58.719460 1717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:58:58.719616 1717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:58:58.720199 1717 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:58:58.720285 1717 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:58:58.720346 1717 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:58:58.720382 1717 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:58:58.727240 1717 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40105
I20260924 04:58:58.727530 1848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40105 every 8 connection(s)
I20260924 04:58:58.727643 1717 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225929848825-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:58:58.730726 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 1717
I20260924 04:58:58.730834 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225929848825-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:58:58.731004 392 external_mini_cluster.cc:1428] Setting key 193ce8b3b45b4d8dbffaaa5d34df6de6
Sep 24 04:58:58 dist-test-slave-70st krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225938, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:58:58.737614 1742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:58:58.720886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46435 (local address 127.0.98.62:42079)
0924 04:58:58.729490 (+ 8604us) server_negotiation.cc:206] Beginning negotiation
0924 04:58:58.729492 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:58:58.729503 (+ 11us) server_negotiation.cc:407] Connection header received
0924 04:58:58.729533 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:58:58.729534 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:58:58.729544 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:58:58.729617 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:58:58.730461 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:58:58.731228 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:58:58.732310 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:58:58.732521 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:58:58.732577 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:58:58.734979 (+ 2402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:58:58.734994 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:58:58.734994 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:58:58.735002 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:58:58.736327 (+ 1325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:58:58.736691 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:58:58.736692 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:58:58.736692 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:58:58.736747 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:58:58.736959 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:58:58.736960 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:58:58.736961 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:58:58.737071 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:58:58.737140 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 04:58:58.737394 (+ 254us) server_negotiation.cc:299] Negotiation successful
0924 04:58:58.737515 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8578,"thread_start_us":48,"threads_started":1}
I20260924 04:58:58.737764 1741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:58:58.720602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42079 (local address 127.0.98.2:46435)
0924 04:58:58.720802 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0924 04:58:58.720850 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 04:58:58.721541 (+ 691us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:58:58.729710 (+ 8169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:58:58.729715 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:58:58.730082 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:58:58.730372 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:58:58.730380 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:58:58.731319 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:58:58.731320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:58:58.732229 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:58:58.732232 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:58:58.732601 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:58:58.732602 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:58:58.732718 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:58:58.733241 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:58:58.733252 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:58:58.734871 (+ 1619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:58:58.736403 (+ 1532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:58:58.736410 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:58:58.736412 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:58:58.736624 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:58:58.736800 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:58:58.736801 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:58:58.736802 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:58:58.736905 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:58:58.737134 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:58:58.737140 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:58:58.737277 (+ 137us) client_negotiation.cc:769] Sending connection context
0924 04:58:58.737364 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 04:58:58.737550 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":71,"threads_started":1}
I20260924 04:58:58.738292 1849 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42079
I20260924 04:58:58.738379 1849 heartbeater.cc:464] Registering TS with master...
I20260924 04:58:58.738579 1849 heartbeater.cc:511] Master 127.0.98.62:42079 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 04:58:58.738929 1504 ts_manager.cc:194] Registered new tserver with Master: 3856444a605042b69a7e3159ec5e35b0 (127.0.98.2:40105)
I20260924 04:58:58.739372 1504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:46435
I20260924 04:58:58.744730 1742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:58:58.739342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56935 (local address 127.0.98.62:42079)
0924 04:58:58.739504 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 04:58:58.739506 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:58:58.739513 (+ 7us) server_negotiation.cc:407] Connection header received
0924 04:58:58.739583 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:58:58.739584 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:58:58.739593 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:58:58.739633 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:58:58.740008 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:58:58.740545 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:58:58.741528 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:58:58.741712 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:58:58.741762 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:58:58.742541 (+ 779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:58:58.742542 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:58:58.742543 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:58:58.742549 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:58:58.743755 (+ 1206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:58:58.744092 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:58:58.744092 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:58:58.744092 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:58:58.744134 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:58:58.744305 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:58:58.744306 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:58:58.744306 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:58:58.744457 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:58:58.744514 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 04:58:58.744611 (+ 97us) server_negotiation.cc:299] Negotiation successful
0924 04:58:58.744671 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142}
I20260924 04:58:58.744735 1856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:58:58.739325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42079 (local address 127.0.98.2:56935)
0924 04:58:58.739460 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 04:58:58.739471 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 04:58:58.739527 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:58:58.739648 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:58:58.739649 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:58:58.739855 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:58:58.739924 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:58:58.739926 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:58:58.740626 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:58:58.740627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:58:58.741461 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:58:58.741463 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:58:58.741766 (+ 303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:58:58.741767 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:58:58.741818 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:58:58.742218 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:58:58.742222 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:58:58.742475 (+ 253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:58:58.743831 (+ 1356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:58:58.743832 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:58:58.743832 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:58:58.744030 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:58:58.744185 (+ 155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:58:58.744186 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:58:58.744186 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:58:58.744244 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:58:58.744516 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:58:58.744517 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:58:58.744548 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 04:58:58.744595 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 04:58:58.744665 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":78,"threads_started":1}
I20260924 04:58:58.749248 1708 heartbeater.cc:503] Master 127.0.98.62:42079 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225929848825-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225929848825-392-0/krb5kdc/kudu_127.0.98.3.keytab.
W20260924 04:58:59.303440 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.3: real 0.572s user 0.000s sys 0.003s
I20260924 04:58:59.303766 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225929848825-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225929848825-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225929848825-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225929848825-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42079
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225929848825-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:58:59.381045 1858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:58:59.381227 1858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:58:59.381247 1858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:58:59.382364 1858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:58:59.382405 1858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:58:59.382423 1858 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 04:58:59.382439 1858 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:58:59.383876 1858 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225929848825-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225929848825-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225929848825-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225929848825-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:42079
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1858
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225929848825-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 04:58:59.384069 1858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:58:59.384268 1858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:58:59.388964 1864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:58:59.388978 1863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:58:59.389040 1858 server_base.cc:1034] running on GCE node
W20260924 04:58:59.389009 1866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:58:59.389313 1858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:58:59.389533 1858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:58:59.390664 1858 hybrid_clock.cc:648] HybridClock initialized: now 1790225939390653 us; error 27 us; skew 500 ppm
Sep 24 04:58:59 dist-test-slave-70st krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225939, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:58:59.392880 1858 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:58:59.393266 1858 webserver.cc:460] Webserver started at http://127.0.98.3:42543/ using document root <none> and password file <none>
I20260924 04:58:59.393476 1858 fs_manager.cc:359] Metadata directory not provided
I20260924 04:58:59.393519 1858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:58:59.393610 1858 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:58:59.394562 1858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225929848825-392-0/minicluster-data/ts-2/data/instance:
uuid: "b0d906902dcd4df68b5f6f3d2de3eb88"
format_stamp: "Formatted at 2026-09-24 04:58:59 on dist-test-slave-70st"
server_key: "cc36f19330aa7e71df9452a2c8365116"
server_key_iv: "51de2af17d510d8414f61029e0f630c0"
server_key_version: "encryptionkey@0"
I20260924 04:58:59.394835 1858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225929848825-392-0/minicluster-data/ts-2/wal/instance:
uuid: "b0d906902dcd4df68b5f6f3d2de3eb88"
format_stamp: "Formatted at 2026-09-24 04:58:59 on dist-test-slave-70st"
server_key: "cc36f19330aa7e71df9452a2c8365116"
server_key_iv: "51de2af17d510d8414f61029e0f630c0"
server_key_version: "encryptionkey@0"
I20260924 04:58:59.395932 1858 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 04:58:59.396520 1873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:58:59.396654 1858 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 04:58:59.396724 1858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225929848825-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225929848825-392-0/minicluster-data/ts-2/wal
uuid: "b0d906902dcd4df68b5f6f3d2de3eb88"
format_stamp: "Formatted at 2026-09-24 04:58:59 on dist-test-slave-70st"
server_key: "cc36f19330aa7e71df9452a2c8365116"
server_key_iv: "51de2af17d510d8414f61029e0f630c0"
server_key_version: "encryptionkey@0"
I20260924 04:58:59.396775 1858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225929848825-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225929848825-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225929848825-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:58:59.412993 1858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:58:59.414701 1858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:58:59.414804 1858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:58:59.415278 1858 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:58:59.415319 1858 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:58:59.415342 1858 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:58:59.415357 1858 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:58:59.421310 1858 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37411
I20260924 04:58:59.421355 1989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37411 every 8 connection(s)
Sep 24 04:58:59 dist-test-slave-70st krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225939, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:58:59.421818 1858 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225929848825-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:58:59.424664 1883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:58:59.415761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33207 (local address 127.0.98.62:42079)
0924 04:58:59.415890 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 04:58:59.415892 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:58:59.416020 (+ 128us) server_negotiation.cc:407] Connection header received
0924 04:58:59.416531 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:58:59.416532 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:58:59.416555 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:58:59.416608 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:58:59.417413 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:58:59.418222 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:58:59.419686 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:58:59.419999 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:58:59.420078 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:58:59.422277 (+ 2199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:58:59.422279 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:58:59.422279 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:58:59.422286 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:58:59.423456 (+ 1170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:58:59.423808 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:58:59.423809 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:58:59.423810 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:58:59.423861 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:58:59.424075 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:58:59.424076 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:58:59.424076 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:58:59.424182 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:58:59.424236 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 04:58:59.424487 (+ 251us) server_negotiation.cc:299] Negotiation successful
0924 04:58:59.424570 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":62,"threads_started":1}
I20260924 04:58:59.424790 1882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:58:59.415756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42079 (local address 127.0.98.3:33207)
0924 04:58:59.415897 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 04:58:59.415941 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 04:58:59.416422 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:58:59.416692 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:58:59.416697 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:58:59.417023 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:58:59.417298 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:58:59.417305 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:58:59.418319 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:58:59.418319 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:58:59.419606 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:58:59.419609 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:58:59.420084 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:58:59.420085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:58:59.420211 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:58:59.420694 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:58:59.420703 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:58:59.422190 (+ 1487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:58:59.423538 (+ 1348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:58:59.423544 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:58:59.423546 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:58:59.423744 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:58:59.423913 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:58:59.423913 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:58:59.423913 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:58:59.424021 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:58:59.424233 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:58:59.424239 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:58:59.424367 (+ 128us) client_negotiation.cc:769] Sending connection context
0924 04:58:59.424451 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 04:58:59.424590 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":70,"threads_started":1}
I20260924 04:58:59.425221 1990 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42079
I20260924 04:58:59.425299 1990 heartbeater.cc:464] Registering TS with master...
I20260924 04:58:59.425496 1990 heartbeater.cc:511] Master 127.0.98.62:42079 requested a full tablet report, sending...
I20260924 04:58:59.425853 1504 ts_manager.cc:194] Registered new tserver with Master: b0d906902dcd4df68b5f6f3d2de3eb88 (127.0.98.3:37411)
I20260924 04:58:59.426297 1504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33207
I20260924 04:58:59.428133 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 1858
I20260924 04:58:59.428253 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225929848825-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:58:59.428431 392 external_mini_cluster.cc:1428] Setting key e61cdbb91a80545bf5be7888e21c7b3c
I20260924 04:58:59.429159 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:58:59.431416 1883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:58:59.426173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35135 (local address 127.0.98.62:42079)
0924 04:58:59.426206 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 04:58:59.426208 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:58:59.426277 (+ 69us) server_negotiation.cc:407] Connection header received
0924 04:58:59.426337 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:58:59.426337 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:58:59.426344 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:58:59.426386 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:58:59.426729 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:58:59.427312 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:58:59.427984 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:58:59.428258 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:58:59.428323 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:58:59.429215 (+ 892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:58:59.429216 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:58:59.429217 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:58:59.429223 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:58:59.430531 (+ 1308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:58:59.430809 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:58:59.430810 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:58:59.430810 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:58:59.430868 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:58:59.431043 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:58:59.431043 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:58:59.431044 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:58:59.431135 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:58:59.431180 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 04:58:59.431304 (+ 124us) server_negotiation.cc:299] Negotiation successful
0924 04:58:59.431359 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260924 04:58:59.431596 1996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:58:59.426118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42079 (local address 127.0.98.3:35135)
0924 04:58:59.426231 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0924 04:58:59.426240 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 04:58:59.426287 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:58:59.426393 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:58:59.426394 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:58:59.426618 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:58:59.426665 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:58:59.426666 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:58:59.427391 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:58:59.427392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:58:59.427926 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:58:59.427927 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:58:59.428324 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:58:59.428325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:58:59.428388 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:58:59.428794 (+ 406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:58:59.428799 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:58:59.429142 (+ 343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:58:59.430601 (+ 1459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:58:59.430602 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:58:59.430602 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:58:59.430749 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:58:59.430926 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:58:59.430927 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:58:59.430927 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:58:59.430989 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:58:59.431182 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:58:59.431182 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:58:59.431222 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 04:58:59.431287 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 04:58:59.431383 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":57,"threads_started":1}
Sep 24 04:58:59 dist-test-slave-70st krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225939, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 04:58:59 dist-test-slave-70st krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225939, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:58:59.449872 1883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:58:59.444096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42072 (local address 127.0.98.62:42079)
0924 04:58:59.444134 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 04:58:59.444135 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 04:58:59.444143 (+ 8us) server_negotiation.cc:407] Connection header received
0924 04:58:59.444193 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:58:59.444194 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:58:59.444201 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:58:59.444248 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:58:59.444718 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:58:59.445193 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:58:59.445884 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:58:59.446078 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:58:59.446124 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:58:59.448035 (+ 1911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:58:59.448036 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:58:59.448036 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:58:59.448042 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:58:59.449052 (+ 1010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:58:59.449311 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:58:59.449311 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:58:59.449312 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:58:59.449352 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:58:59.449514 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:58:59.449515 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:58:59.449515 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:58:59.449610 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:58:59.449682 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 04:58:59.449756 (+ 74us) server_negotiation.cc:299] Negotiation successful
0924 04:58:59.449809 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 04:58:59.451318 1504 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42072:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:58:59.452783 1504 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:58:59.458547 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:58:59.454928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44377 (local address 127.0.0.1:40854)
0924 04:58:59.455111 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0924 04:58:59.455119 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 04:58:59.455169 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:58:59.455719 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:58:59.455720 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:58:59.455721 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:58:59.455759 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:58:59.455760 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:58:59.456571 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:58:59.456572 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:58:59.457463 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:58:59.457466 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:58:59.458327 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:58:59.458328 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:58:59.458382 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:58:59.458410 (+ 28us) client_negotiation.cc:769] Sending connection context
0924 04:58:59.458460 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 04:58:59.458494 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"mutex_wait_us":39,"thread_start_us":110,"threads_started":1}
I20260924 04:58:59.458596 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:58:59.455462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40854 (local address 127.0.98.1:44377)
0924 04:58:59.455554 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0924 04:58:59.455557 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:58:59.455568 (+ 11us) server_negotiation.cc:407] Connection header received
0924 04:58:59.455603 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:58:59.455605 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:58:59.455622 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:58:59.455721 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:58:59.455816 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:58:59.456495 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:58:59.457541 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:58:59.458247 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:58:59.458315 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:58:59.458371 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 04:58:59.458492 (+ 121us) server_negotiation.cc:299] Negotiation successful
0924 04:58:59.458528 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70,"thread_start_us":48,"threads_started":1}
I20260924 04:58:59.459086 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:58:59.454918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40105 (local address 127.0.0.1:37684)
0924 04:58:59.455070 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 04:58:59.455108 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 04:58:59.455169 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:58:59.455445 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:58:59.455447 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:58:59.455450 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:58:59.455505 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:58:59.455507 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:58:59.456550 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:58:59.456551 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:58:59.457832 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:58:59.457834 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:58:59.458850 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:58:59.458851 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:58:59.458928 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:58:59.458932 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 04:58:59.458990 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 04:58:59.459026 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"mutex_wait_us":30,"thread_start_us":103,"threads_started":1}
I20260924 04:58:59.459137 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:58:59.455205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37684 (local address 127.0.98.2:40105)
0924 04:58:59.455311 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0924 04:58:59.455316 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:58:59.455323 (+ 7us) server_negotiation.cc:407] Connection header received
0924 04:58:59.455349 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:58:59.455351 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:58:59.455363 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:58:59.455432 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:58:59.455555 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:58:59.456478 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:58:59.457900 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:58:59.458559 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:58:59.458620 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:58:59.458672 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 04:58:59.459017 (+ 345us) server_negotiation.cc:299] Negotiation successful
0924 04:58:59.459059 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78,"thread_start_us":52,"threads_started":1}
I20260924 04:58:59.459268 1642 tablet_service.cc:1467] Processing CreateTablet for tablet ccf6a5196af7400d9a245e4dc2a109b3 (DEFAULT_TABLE table=test-table [id=671282291b1546d091d22a72fd56de21]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:58:59.459631 1642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccf6a5196af7400d9a245e4dc2a109b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:58:59.459971 1783 tablet_service.cc:1467] Processing CreateTablet for tablet ccf6a5196af7400d9a245e4dc2a109b3 (DEFAULT_TABLE table=test-table [id=671282291b1546d091d22a72fd56de21]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:58:59.460296 1783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccf6a5196af7400d9a245e4dc2a109b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:58:59.460753 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:58:59.454929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37411 (local address 127.0.0.1:51944)
0924 04:58:59.455589 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0924 04:58:59.455599 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 04:58:59.455661 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:58:59.456581 (+ 920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:58:59.456581 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:58:59.456584 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:58:59.456616 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:58:59.456617 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:58:59.458727 (+ 2110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:58:59.458728 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:58:59.459670 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:58:59.459672 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:58:59.460520 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:58:59.460521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:58:59.460584 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:58:59.460588 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 04:58:59.460657 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 04:58:59.460691 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"thread_start_us":98,"threads_started":1}
I20260924 04:58:59.460793 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:58:59.455319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51944 (local address 127.0.98.3:37411)
0924 04:58:59.455453 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 04:58:59.455458 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:58:59.455671 (+ 213us) server_negotiation.cc:407] Connection header received
0924 04:58:59.455728 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:58:59.455730 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:58:59.455746 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:58:59.455815 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:58:59.456685 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:58:59.457835 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:58:59.459775 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:58:59.460440 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:58:59.460513 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:58:59.460585 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 04:58:59.460671 (+ 86us) server_negotiation.cc:299] Negotiation successful
0924 04:58:59.460710 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":60,"threads_started":1}
I20260924 04:58:59.462061 1924 tablet_service.cc:1467] Processing CreateTablet for tablet ccf6a5196af7400d9a245e4dc2a109b3 (DEFAULT_TABLE table=test-table [id=671282291b1546d091d22a72fd56de21]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:58:59.462381 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccf6a5196af7400d9a245e4dc2a109b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:58:59.462915 2015 tablet_bootstrap.cc:492] T ccf6a5196af7400d9a245e4dc2a109b3 P 3856444a605042b69a7e3159ec5e35b0: Bootstrap starting.
I20260924 04:58:59.462800 2014 tablet_bootstrap.cc:492] T ccf6a5196af7400d9a245e4dc2a109b3 P 239a221e14974ed1be41fa89fe85928f: Bootstrap starting.
I20260924 04:58:59.463450 2014 tablet_bootstrap.cc:654] T ccf6a5196af7400d9a245e4dc2a109b3 P 239a221e14974ed1be41fa89fe85928f: Neither blocks nor log segments found. Creating new log.
I20260924 04:58:59.463522 2015 tablet_bootstrap.cc:654] T ccf6a5196af7400d9a245e4dc2a109b3 P 3856444a605042b69a7e3159ec5e35b0: Neither blocks nor log segments found. Creating new log.
I20260924 04:58:59.463716 2014 log.cc:824] T ccf6a5196af7400d9a245e4dc2a109b3 P 239a221e14974ed1be41fa89fe85928f: Log is configured to *not* fsync() on all Append() calls
I20260924 04:58:59.463810 2015 log.cc:824] T ccf6a5196af7400d9a245e4dc2a109b3 P 3856444a605042b69a7e3159ec5e35b0: Log is configured to *not* fsync() on all Append() calls
I20260924 04:58:59.464455 2014 tablet_bootstrap.cc:492] T ccf6a5196af7400d9a245e4dc2a109b3 P 239a221e14974ed1be41fa89fe85928f: No bootstrap required, opened a new log
I20260924 04:58:59.464455 2015 tablet_bootstrap.cc:492] T ccf6a5196af7400d9a245e4dc2a109b3 P 3856444a605042b69a7e3159ec5e35b0: No bootstrap required, opened a new log
I20260924 04:58:59.464525 2015 ts_tablet_manager.cc:1397] T ccf6a5196af7400d9a245e4dc2a109b3 P 3856444a605042b69a7e3159ec5e35b0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 04:58:59.464525 2014 ts_tablet_manager.cc:1397] T ccf6a5196af7400d9a245e4dc2a109b3 P 239a221e14974ed1be41fa89fe85928f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 04:58:59.465401 2018 tablet_bootstrap.cc:492] T ccf6a5196af7400d9a245e4dc2a109b3 P b0d906902dcd4df68b5f6f3d2de3eb88: Bootstrap starting.
I20260924 04:58:59.465960 2018 tablet_bootstrap.cc:654] T ccf6a5196af7400d9a245e4dc2a109b3 P b0d906902dcd4df68b5f6f3d2de3eb88: Neither blocks nor log segments found. Creating new log.
I20260924 04:58:59.466239 2018 log.cc:824] T ccf6a5196af7400d9a245e4dc2a109b3 P b0d906902dcd4df68b5f6f3d2de3eb88: Log is configured to *not* fsync() on all Append() calls
I20260924 04:58:59.466780 2018 tablet_bootstrap.cc:492] T ccf6a5196af7400d9a245e4dc2a109b3 P b0d906902dcd4df68b5f6f3d2de3eb88: No bootstrap required, opened a new log
I20260924 04:58:59.466831 2018 ts_tablet_manager.cc:1397] T ccf6a5196af7400d9a245e4dc2a109b3 P b0d906902dcd4df68b5f6f3d2de3eb88: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260924 04:58:59.466941 2015 raft_consensus.cc:348] T ccf6a5196af7400d9a245e4dc2a109b3 P 3856444a605042b69a7e3159ec5e35b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3856444a605042b69a7e3159ec5e35b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40105 } } peers { permanent_uuid: "239a221e14974ed1be41fa89fe85928f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44377 } } peers { permanent_uuid: "b0d906902dcd4df68b5f6f3d2de3eb88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37411 } }
I20260924 04:58:59.467103 2015 raft_consensus.cc:374] T ccf6a5196af7400d9a245e4dc2a109b3 P 3856444a605042b69a7e3159ec5e35b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:58:59.467139 2015 raft_consensus.cc:729] T ccf6a5196af7400d9a245e4dc2a109b3 P 3856444a605042b69a7e3159ec5e35b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3856444a605042b69a7e3159ec5e35b0, State: Initialized, Role: FOLLOWER
I20260924 04:58:59.467180 2014 raft_consensus.cc:348] T ccf6a5196af7400d9a245e4dc2a109b3 P 239a221e14974ed1be41fa89fe85928f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3856444a605042b69a7e3159ec5e35b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40105 } } peers { permanent_uuid: "239a221e14974ed1be41fa89fe85928f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44377 } } peers { permanent_uuid: "b0d906902dcd4df68b5f6f3d2de3eb88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37411 } }
I20260924 04:58:59.467249 2015 consensus_queue.cc:260] T ccf6a5196af7400d9a245e4dc2a109b3 P 3856444a605042b69a7e3159ec5e35b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3856444a605042b69a7e3159ec5e35b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40105 } } peers { permanent_uuid: "239a221e14974ed1be41fa89fe85928f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44377 } } peers { permanent_uuid: "b0d906902dcd4df68b5f6f3d2de3eb88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37411 } }
I20260924 04:58:59.467315 2014 raft_consensus.cc:374] T ccf6a5196af7400d9a245e4dc2a109b3 P 239a221e14974ed1be41fa89fe85928f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:58:59.467352 2014 raft_consensus.cc:729] T ccf6a5196af7400d9a245e4dc2a109b3 P 239a221e14974ed1be41fa89fe85928f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 239a221e14974ed1be41fa89fe85928f, State: Initialized, Role: FOLLOWER
I20260924 04:58:59.467455 2015 ts_tablet_manager.cc:1428] T ccf6a5196af7400d9a245e4dc2a109b3 P 3856444a605042b69a7e3159ec5e35b0: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260924 04:58:59.467465 2014 consensus_queue.cc:260] T ccf6a5196af7400d9a245e4dc2a109b3 P 239a221e14974ed1be41fa89fe85928f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3856444a605042b69a7e3159ec5e35b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40105 } } peers { permanent_uuid: "239a221e14974ed1be41fa89fe85928f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44377 } } peers { permanent_uuid: "b0d906902dcd4df68b5f6f3d2de3eb88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37411 } }
I20260924 04:58:59.467577 1849 heartbeater.cc:503] Master 127.0.98.62:42079 was elected leader, sending a full tablet report...
I20260924 04:58:59.467808 2014 ts_tablet_manager.cc:1428] T ccf6a5196af7400d9a245e4dc2a109b3 P 239a221e14974ed1be41fa89fe85928f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 04:58:59.468895 2018 raft_consensus.cc:348] T ccf6a5196af7400d9a245e4dc2a109b3 P b0d906902dcd4df68b5f6f3d2de3eb88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3856444a605042b69a7e3159ec5e35b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40105 } } peers { permanent_uuid: "239a221e14974ed1be41fa89fe85928f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44377 } } peers { permanent_uuid: "b0d906902dcd4df68b5f6f3d2de3eb88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37411 } }
I20260924 04:58:59.469012 2018 raft_consensus.cc:374] T ccf6a5196af7400d9a245e4dc2a109b3 P b0d906902dcd4df68b5f6f3d2de3eb88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:58:59.469040 2018 raft_consensus.cc:729] T ccf6a5196af7400d9a245e4dc2a109b3 P b0d906902dcd4df68b5f6f3d2de3eb88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0d906902dcd4df68b5f6f3d2de3eb88, State: Initialized, Role: FOLLOWER
I20260924 04:58:59.469141 2018 consensus_queue.cc:260] T ccf6a5196af7400d9a245e4dc2a109b3 P b0d906902dcd4df68b5f6f3d2de3eb88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3856444a605042b69a7e3159ec5e35b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40105 } } peers { permanent_uuid: "239a221e14974ed1be41fa89fe85928f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44377 } } peers { permanent_uuid: "b0d906902dcd4df68b5f6f3d2de3eb88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37411 } }
I20260924 04:58:59.469332 2018 ts_tablet_manager.cc:1428] T ccf6a5196af7400d9a245e4dc2a109b3 P b0d906902dcd4df68b5f6f3d2de3eb88: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 04:58:59.469388 1990 heartbeater.cc:503] Master 127.0.98.62:42079 was elected leader, sending a full tablet report...
W20260924 04:58:59.478451 1850 tablet.cc:2369] T ccf6a5196af7400d9a245e4dc2a109b3 P 3856444a605042b69a7e3159ec5e35b0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:58:59.487396 2022 raft_consensus.cc:482] T ccf6a5196af7400d9a245e4dc2a109b3 P b0d906902dcd4df68b5f6f3d2de3eb88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:58:59.487478 2022 raft_consensus.cc:504] T ccf6a5196af7400d9a245e4dc2a109b3 P b0d906902dcd4df68b5f6f3d2de3eb88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3856444a605042b69a7e3159ec5e35b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40105 } } peers { permanent_uuid: "239a221e14974ed1be41fa89fe85928f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44377 } } peers { permanent_uuid: "b0d906902dcd4df68b5f6f3d2de3eb88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37411 } }
I20260924 04:58:59.487733 2022 leader_election.cc:288] T ccf6a5196af7400d9a245e4dc2a109b3 P b0d906902dcd4df68b5f6f3d2de3eb88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3856444a605042b69a7e3159ec5e35b0 (127.0.98.2:40105), 239a221e14974ed1be41fa89fe85928f (127.0.98.1:44377)
I20260924 04:58:59.490315 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:58:59.487981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60683 (local address 127.0.98.1:44377)
0924 04:58:59.488030 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 04:58:59.488031 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 04:58:59.488038 (+ 7us) server_negotiation.cc:407] Connection header received
0924 04:58:59.488061 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:58:59.488062 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:58:59.488073 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:58:59.488119 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:58:59.488246 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:58:59.488775 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:58:59.489679 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:58:59.490023 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:58:59.490082 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:58:59.490092 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 04:58:59.490222 (+ 130us) server_negotiation.cc:299] Negotiation successful
0924 04:58:59.490255 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 04:58:59.490325 1882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:58:59.487889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44377 (local address 127.0.98.3:60683)
0924 04:58:59.487958 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0924 04:58:59.487970 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:58:59.488017 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:58:59.488125 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:58:59.488126 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:58:59.488127 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:58:59.488188 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:58:59.488190 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:58:59.488841 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:58:59.488842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:58:59.489607 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:58:59.489610 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:58:59.490086 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:58:59.490087 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:58:59.490157 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:58:59.490160 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 04:58:59.490208 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 04:58:59.490244 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260924 04:58:59.490366 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:58:59.487949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59995 (local address 127.0.98.2:40105)
0924 04:58:59.487987 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 04:58:59.487988 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 04:58:59.488130 (+ 142us) server_negotiation.cc:407] Connection header received
0924 04:58:59.488180 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:58:59.488181 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:58:59.488190 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:58:59.488236 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:58:59.488365 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:58:59.488836 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:58:59.489720 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:58:59.490073 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:58:59.490131 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:58:59.490140 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 04:58:59.490274 (+ 134us) server_negotiation.cc:299] Negotiation successful
0924 04:58:59.490308 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 04:58:59.490402 2023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:58:59.487894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40105 (local address 127.0.98.3:59995)
0924 04:58:59.488062 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0924 04:58:59.488072 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 04:58:59.488126 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:58:59.488244 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:58:59.488244 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:58:59.488246 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:58:59.488314 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:58:59.488316 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:58:59.488901 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:58:59.488902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:58:59.489641 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:58:59.489643 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:58:59.490140 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:58:59.490141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:58:59.490215 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:58:59.490218 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 04:58:59.490261 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 04:58:59.490297 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"mutex_wait_us":11,"thread_start_us":114,"threads_started":1}
I20260924 04:58:59.490694 1803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccf6a5196af7400d9a245e4dc2a109b3" candidate_uuid: "b0d906902dcd4df68b5f6f3d2de3eb88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3856444a605042b69a7e3159ec5e35b0" is_pre_election: true
I20260924 04:58:59.490818 1803 raft_consensus.cc:2454] T ccf6a5196af7400d9a245e4dc2a109b3 P 3856444a605042b69a7e3159ec5e35b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0d906902dcd4df68b5f6f3d2de3eb88 in term 0.
I20260924 04:58:59.490798 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccf6a5196af7400d9a245e4dc2a109b3" candidate_uuid: "b0d906902dcd4df68b5f6f3d2de3eb88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "239a221e14974ed1be41fa89fe85928f" is_pre_election: true
I20260924 04:58:59.490932 1662 raft_consensus.cc:2454] T ccf6a5196af7400d9a245e4dc2a109b3 P 239a221e14974ed1be41fa89fe85928f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0d906902dcd4df68b5f6f3d2de3eb88 in term 0.
I20260924 04:58:59.491057 1877 leader_election.cc:302] T ccf6a5196af7400d9a245e4dc2a109b3 P b0d906902dcd4df68b5f6f3d2de3eb88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3856444a605042b69a7e3159ec5e35b0, b0d906902dcd4df68b5f6f3d2de3eb88; no voters:
I20260924 04:58:59.491228 2022 raft_consensus.cc:2781] T ccf6a5196af7400d9a245e4dc2a109b3 P b0d906902dcd4df68b5f6f3d2de3eb88 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:58:59.491269 2022 raft_consensus.cc:482] T ccf6a5196af7400d9a245e4dc2a109b3 P b0d906902dcd4df68b5f6f3d2de3eb88 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:58:59.491284 2022 raft_consensus.cc:3037] T ccf6a5196af7400d9a245e4dc2a109b3 P b0d906902dcd4df68b5f6f3d2de3eb88 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:58:59.492130 2022 raft_consensus.cc:504] T ccf6a5196af7400d9a245e4dc2a109b3 P b0d906902dcd4df68b5f6f3d2de3eb88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3856444a605042b69a7e3159ec5e35b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40105 } } peers { permanent_uuid: "239a221e14974ed1be41fa89fe85928f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44377 } } peers { permanent_uuid: "b0d906902dcd4df68b5f6f3d2de3eb88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37411 } }
I20260924 04:58:59.492247 2022 leader_election.cc:288] T ccf6a5196af7400d9a245e4dc2a109b3 P b0d906902dcd4df68b5f6f3d2de3eb88 [CANDIDATE]: Term 1 election: Requested vote from peers 3856444a605042b69a7e3159ec5e35b0 (127.0.98.2:40105), 239a221e14974ed1be41fa89fe85928f (127.0.98.1:44377)
I20260924 04:58:59.492388 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccf6a5196af7400d9a245e4dc2a109b3" candidate_uuid: "b0d906902dcd4df68b5f6f3d2de3eb88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "239a221e14974ed1be41fa89fe85928f"
I20260924 04:58:59.492378 1803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccf6a5196af7400d9a245e4dc2a109b3" candidate_uuid: "b0d906902dcd4df68b5f6f3d2de3eb88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3856444a605042b69a7e3159ec5e35b0"
I20260924 04:58:59.492455 1662 raft_consensus.cc:3037] T ccf6a5196af7400d9a245e4dc2a109b3 P 239a221e14974ed1be41fa89fe85928f [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:58:59.492456 1803 raft_consensus.cc:3037] T ccf6a5196af7400d9a245e4dc2a109b3 P 3856444a605042b69a7e3159ec5e35b0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:58:59.493067 1803 raft_consensus.cc:2454] T ccf6a5196af7400d9a245e4dc2a109b3 P 3856444a605042b69a7e3159ec5e35b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0d906902dcd4df68b5f6f3d2de3eb88 in term 1.
I20260924 04:58:59.493108 1662 raft_consensus.cc:2454] T ccf6a5196af7400d9a245e4dc2a109b3 P 239a221e14974ed1be41fa89fe85928f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0d906902dcd4df68b5f6f3d2de3eb88 in term 1.
I20260924 04:58:59.493219 1877 leader_election.cc:302] T ccf6a5196af7400d9a245e4dc2a109b3 P b0d906902dcd4df68b5f6f3d2de3eb88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3856444a605042b69a7e3159ec5e35b0, b0d906902dcd4df68b5f6f3d2de3eb88; no voters:
I20260924 04:58:59.493290 2022 raft_consensus.cc:2781] T ccf6a5196af7400d9a245e4dc2a109b3 P b0d906902dcd4df68b5f6f3d2de3eb88 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:58:59.493438 2022 raft_consensus.cc:686] T ccf6a5196af7400d9a245e4dc2a109b3 P b0d906902dcd4df68b5f6f3d2de3eb88 [term 1 LEADER]: Becoming Leader. State: Replica: b0d906902dcd4df68b5f6f3d2de3eb88, State: Running, Role: LEADER
I20260924 04:58:59.493520 2022 consensus_queue.cc:237] T ccf6a5196af7400d9a245e4dc2a109b3 P b0d906902dcd4df68b5f6f3d2de3eb88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3856444a605042b69a7e3159ec5e35b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40105 } } peers { permanent_uuid: "239a221e14974ed1be41fa89fe85928f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44377 } } peers { permanent_uuid: "b0d906902dcd4df68b5f6f3d2de3eb88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37411 } }
I20260924 04:58:59.494225 1502 catalog_manager.cc:5495] T ccf6a5196af7400d9a245e4dc2a109b3 P b0d906902dcd4df68b5f6f3d2de3eb88 reported cstate change: term changed from 0 to 1, leader changed from <none> to b0d906902dcd4df68b5f6f3d2de3eb88 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "b0d906902dcd4df68b5f6f3d2de3eb88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3856444a605042b69a7e3159ec5e35b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40105 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "239a221e14974ed1be41fa89fe85928f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b0d906902dcd4df68b5f6f3d2de3eb88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37411 } health_report { overall_health: HEALTHY } } }
W20260924 04:58:59.494937 1709 tablet.cc:2369] T ccf6a5196af7400d9a245e4dc2a109b3 P 239a221e14974ed1be41fa89fe85928f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:58:59.505168 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:58:59.503078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51958 (local address 127.0.98.3:37411)
0924 04:58:59.503122 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 04:58:59.503123 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 04:58:59.503143 (+ 20us) server_negotiation.cc:407] Connection header received
0924 04:58:59.503221 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:58:59.503222 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:58:59.503235 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:58:59.503280 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:58:59.503376 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:58:59.503820 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:58:59.504580 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:58:59.504753 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:58:59.504799 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:58:59.504941 (+ 142us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:58:59.505028 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:58:59.505078 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 04:58:59.505108 (+ 30us) server_negotiation.cc:299] Negotiation successful
0924 04:58:59.505121 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 04:58:59.506739 1662 raft_consensus.cc:1261] T ccf6a5196af7400d9a245e4dc2a109b3 P 239a221e14974ed1be41fa89fe85928f [term 1 FOLLOWER]: Refusing update from remote peer b0d906902dcd4df68b5f6f3d2de3eb88: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:58:59.506758 1803 raft_consensus.cc:1261] T ccf6a5196af7400d9a245e4dc2a109b3 P 3856444a605042b69a7e3159ec5e35b0 [term 1 FOLLOWER]: Refusing update from remote peer b0d906902dcd4df68b5f6f3d2de3eb88: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:58:59.507061 2024 consensus_queue.cc:1035] T ccf6a5196af7400d9a245e4dc2a109b3 P b0d906902dcd4df68b5f6f3d2de3eb88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3856444a605042b69a7e3159ec5e35b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:58:59.507164 2022 consensus_queue.cc:1035] T ccf6a5196af7400d9a245e4dc2a109b3 P b0d906902dcd4df68b5f6f3d2de3eb88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "239a221e14974ed1be41fa89fe85928f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:58:59.509485 2027 mvcc.cc:205] Tried to move back new op lower bound from 7332765448217362432 to 7332765448166072320. Current Snapshot: MvccSnapshot[applied={T|T < 7332765448217362432}]
I20260924 04:58:59.509974 2028 mvcc.cc:205] Tried to move back new op lower bound from 7332765448217362432 to 7332765448166072320. Current Snapshot: MvccSnapshot[applied={T|T < 7332765448217362432}]
I20260924 04:58:59.512790 1502 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42072:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:58:59.512869 1502 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42072:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:58:59.513518 1502 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d11579be4182427e90eaae6d4647dff1: Sending DeleteTablet for 3 replicas of tablet ccf6a5196af7400d9a245e4dc2a109b3
I20260924 04:58:59.513811 1924 tablet_service.cc:1514] Processing DeleteTablet for tablet ccf6a5196af7400d9a245e4dc2a109b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:58:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51944
I20260924 04:58:59.513830 1783 tablet_service.cc:1514] Processing DeleteTablet for tablet ccf6a5196af7400d9a245e4dc2a109b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:58:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37684
I20260924 04:58:59.513831 1642 tablet_service.cc:1514] Processing DeleteTablet for tablet ccf6a5196af7400d9a245e4dc2a109b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:58:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40854
I20260924 04:58:59.514045 2037 tablet_replica.cc:330] T ccf6a5196af7400d9a245e4dc2a109b3 P 239a221e14974ed1be41fa89fe85928f: stopping tablet replica
I20260924 04:58:59.514159 2037 raft_consensus.cc:2229] T ccf6a5196af7400d9a245e4dc2a109b3 P 239a221e14974ed1be41fa89fe85928f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:58:59.514261 2037 raft_consensus.cc:2258] T ccf6a5196af7400d9a245e4dc2a109b3 P 239a221e14974ed1be41fa89fe85928f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:58:59.514305 1502 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42072:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:58:59.514432 1502 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:58:59.514681 2037 ts_tablet_manager.cc:1905] T ccf6a5196af7400d9a245e4dc2a109b3 P 239a221e14974ed1be41fa89fe85928f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:58:59.514698 2038 tablet_replica.cc:330] T ccf6a5196af7400d9a245e4dc2a109b3 P 3856444a605042b69a7e3159ec5e35b0: stopping tablet replica
I20260924 04:58:59.514771 2038 raft_consensus.cc:2229] T ccf6a5196af7400d9a245e4dc2a109b3 P 3856444a605042b69a7e3159ec5e35b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:58:59.514895 2038 raft_consensus.cc:2258] T ccf6a5196af7400d9a245e4dc2a109b3 P 3856444a605042b69a7e3159ec5e35b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:58:59.515506 1642 tablet_service.cc:1467] Processing CreateTablet for tablet 9e9ec8ea5f0244478c9e685bdbd9233e (DEFAULT_TABLE table=test-table [id=e755be6f8db147cd9cc4ac0ffd989ba7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:58:59.515537 1783 tablet_service.cc:1467] Processing CreateTablet for tablet 9e9ec8ea5f0244478c9e685bdbd9233e (DEFAULT_TABLE table=test-table [id=e755be6f8db147cd9cc4ac0ffd989ba7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:58:59.515537 1924 tablet_service.cc:1467] Processing CreateTablet for tablet 9e9ec8ea5f0244478c9e685bdbd9233e (DEFAULT_TABLE table=test-table [id=e755be6f8db147cd9cc4ac0ffd989ba7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:58:59.515612 1642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e9ec8ea5f0244478c9e685bdbd9233e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:58:59.515637 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e9ec8ea5f0244478c9e685bdbd9233e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:58:59.515636 1783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e9ec8ea5f0244478c9e685bdbd9233e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:58:59.515687 2038 ts_tablet_manager.cc:1905] T ccf6a5196af7400d9a245e4dc2a109b3 P 3856444a605042b69a7e3159ec5e35b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:58:59.516772 2014 tablet_bootstrap.cc:492] T 9e9ec8ea5f0244478c9e685bdbd9233e P 239a221e14974ed1be41fa89fe85928f: Bootstrap starting.
I20260924 04:58:59.516834 2015 tablet_bootstrap.cc:492] T 9e9ec8ea5f0244478c9e685bdbd9233e P 3856444a605042b69a7e3159ec5e35b0: Bootstrap starting.
I20260924 04:58:59.516996 2018 tablet_bootstrap.cc:492] T 9e9ec8ea5f0244478c9e685bdbd9233e P b0d906902dcd4df68b5f6f3d2de3eb88: Bootstrap starting.
I20260924 04:58:59.517129 2014 tablet_bootstrap.cc:654] T 9e9ec8ea5f0244478c9e685bdbd9233e P 239a221e14974ed1be41fa89fe85928f: Neither blocks nor log segments found. Creating new log.
I20260924 04:58:59.517212 2015 tablet_bootstrap.cc:654] T 9e9ec8ea5f0244478c9e685bdbd9233e P 3856444a605042b69a7e3159ec5e35b0: Neither blocks nor log segments found. Creating new log.
I20260924 04:58:59.517514 2018 tablet_bootstrap.cc:654] T 9e9ec8ea5f0244478c9e685bdbd9233e P b0d906902dcd4df68b5f6f3d2de3eb88: Neither blocks nor log segments found. Creating new log.
I20260924 04:58:59.517863 2038 ts_tablet_manager.cc:1918] T ccf6a5196af7400d9a245e4dc2a109b3 P 3856444a605042b69a7e3159ec5e35b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:58:59.517933 2038 log.cc:1196] T ccf6a5196af7400d9a245e4dc2a109b3 P 3856444a605042b69a7e3159ec5e35b0: Deleting WAL directory at /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225929848825-392-0/minicluster-data/ts-1/wal/wals/ccf6a5196af7400d9a245e4dc2a109b3
I20260924 04:58:59.518191 2038 ts_tablet_manager.cc:1939] T ccf6a5196af7400d9a245e4dc2a109b3 P 3856444a605042b69a7e3159ec5e35b0: Deleting consensus metadata
I20260924 04:58:59.518376 2037 ts_tablet_manager.cc:1918] T ccf6a5196af7400d9a245e4dc2a109b3 P 239a221e14974ed1be41fa89fe85928f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:58:59.518443 2037 log.cc:1196] T ccf6a5196af7400d9a245e4dc2a109b3 P 239a221e14974ed1be41fa89fe85928f: Deleting WAL directory at /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225929848825-392-0/minicluster-data/ts-0/wal/wals/ccf6a5196af7400d9a245e4dc2a109b3
I20260924 04:58:59.518680 2037 ts_tablet_manager.cc:1939] T ccf6a5196af7400d9a245e4dc2a109b3 P 239a221e14974ed1be41fa89fe85928f: Deleting consensus metadata
I20260924 04:58:59.518898 1491 catalog_manager.cc:4841] TS 3856444a605042b69a7e3159ec5e35b0 (127.0.98.2:40105): tablet ccf6a5196af7400d9a245e4dc2a109b3 (table test-table [id=671282291b1546d091d22a72fd56de21]) successfully deleted
I20260924 04:58:59.519049 2014 tablet_bootstrap.cc:492] T 9e9ec8ea5f0244478c9e685bdbd9233e P 239a221e14974ed1be41fa89fe85928f: No bootstrap required, opened a new log
I20260924 04:58:59.519094 2014 ts_tablet_manager.cc:1397] T 9e9ec8ea5f0244478c9e685bdbd9233e P 239a221e14974ed1be41fa89fe85928f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 04:58:59.519343 2018 tablet_bootstrap.cc:492] T 9e9ec8ea5f0244478c9e685bdbd9233e P b0d906902dcd4df68b5f6f3d2de3eb88: No bootstrap required, opened a new log
I20260924 04:58:59.519320 2014 raft_consensus.cc:348] T 9e9ec8ea5f0244478c9e685bdbd9233e P 239a221e14974ed1be41fa89fe85928f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "239a221e14974ed1be41fa89fe85928f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44377 } } peers { permanent_uuid: "3856444a605042b69a7e3159ec5e35b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40105 } } peers { permanent_uuid: "b0d906902dcd4df68b5f6f3d2de3eb88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37411 } }
I20260924 04:58:59.519383 2014 raft_consensus.cc:374] T 9e9ec8ea5f0244478c9e685bdbd9233e P 239a221e14974ed1be41fa89fe85928f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:58:59.519400 2014 raft_consensus.cc:729] T 9e9ec8ea5f0244478c9e685bdbd9233e P 239a221e14974ed1be41fa89fe85928f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 239a221e14974ed1be41fa89fe85928f, State: Initialized, Role: FOLLOWER
I20260924 04:58:59.519403 2018 ts_tablet_manager.cc:1397] T 9e9ec8ea5f0244478c9e685bdbd9233e P b0d906902dcd4df68b5f6f3d2de3eb88: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 04:58:59.519428 2014 consensus_queue.cc:260] T 9e9ec8ea5f0244478c9e685bdbd9233e P 239a221e14974ed1be41fa89fe85928f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "239a221e14974ed1be41fa89fe85928f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44377 } } peers { permanent_uuid: "3856444a605042b69a7e3159ec5e35b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40105 } } peers { permanent_uuid: "b0d906902dcd4df68b5f6f3d2de3eb88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37411 } }
I20260924 04:58:59.519533 2014 ts_tablet_manager.cc:1428] T 9e9ec8ea5f0244478c9e685bdbd9233e P 239a221e14974ed1be41fa89fe85928f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 04:58:59.519533 2018 raft_consensus.cc:348] T 9e9ec8ea5f0244478c9e685bdbd9233e P b0d906902dcd4df68b5f6f3d2de3eb88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "239a221e14974ed1be41fa89fe85928f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44377 } } peers { permanent_uuid: "3856444a605042b69a7e3159ec5e35b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40105 } } peers { permanent_uuid: "b0d906902dcd4df68b5f6f3d2de3eb88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37411 } }
I20260924 04:58:59.519601 2018 raft_consensus.cc:374] T 9e9ec8ea5f0244478c9e685bdbd9233e P b0d906902dcd4df68b5f6f3d2de3eb88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:58:59.519629 2018 raft_consensus.cc:729] T 9e9ec8ea5f0244478c9e685bdbd9233e P b0d906902dcd4df68b5f6f3d2de3eb88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0d906902dcd4df68b5f6f3d2de3eb88, State: Initialized, Role: FOLLOWER
I20260924 04:58:59.519693 2018 consensus_queue.cc:260] T 9e9ec8ea5f0244478c9e685bdbd9233e P b0d906902dcd4df68b5f6f3d2de3eb88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "239a221e14974ed1be41fa89fe85928f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44377 } } peers { permanent_uuid: "3856444a605042b69a7e3159ec5e35b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40105 } } peers { permanent_uuid: "b0d906902dcd4df68b5f6f3d2de3eb88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37411 } }
I20260924 04:58:59.519774 2018 ts_tablet_manager.cc:1428] T 9e9ec8ea5f0244478c9e685bdbd9233e P b0d906902dcd4df68b5f6f3d2de3eb88: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 04:58:59.519763 1489 catalog_manager.cc:4841] TS 239a221e14974ed1be41fa89fe85928f (127.0.98.1:44377): tablet ccf6a5196af7400d9a245e4dc2a109b3 (table test-table [id=671282291b1546d091d22a72fd56de21]) successfully deleted
I20260924 04:58:59.521724 2036 tablet_replica.cc:330] T ccf6a5196af7400d9a245e4dc2a109b3 P b0d906902dcd4df68b5f6f3d2de3eb88: stopping tablet replica
I20260924 04:58:59.521798 2015 tablet_bootstrap.cc:492] T 9e9ec8ea5f0244478c9e685bdbd9233e P 3856444a605042b69a7e3159ec5e35b0: No bootstrap required, opened a new log
I20260924 04:58:59.521843 2015 ts_tablet_manager.cc:1397] T 9e9ec8ea5f0244478c9e685bdbd9233e P 3856444a605042b69a7e3159ec5e35b0: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260924 04:58:59.521945 2036 raft_consensus.cc:2229] T ccf6a5196af7400d9a245e4dc2a109b3 P b0d906902dcd4df68b5f6f3d2de3eb88 [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:58:59.521981 2015 raft_consensus.cc:348] T 9e9ec8ea5f0244478c9e685bdbd9233e P 3856444a605042b69a7e3159ec5e35b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "239a221e14974ed1be41fa89fe85928f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44377 } } peers { permanent_uuid: "3856444a605042b69a7e3159ec5e35b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40105 } } peers { permanent_uuid: "b0d906902dcd4df68b5f6f3d2de3eb88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37411 } }
I20260924 04:58:59.522045 2015 raft_consensus.cc:374] T 9e9ec8ea5f0244478c9e685bdbd9233e P 3856444a605042b69a7e3159ec5e35b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:58:59.522066 2015 raft_consensus.cc:729] T 9e9ec8ea5f0244478c9e685bdbd9233e P 3856444a605042b69a7e3159ec5e35b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3856444a605042b69a7e3159ec5e35b0, State: Initialized, Role: FOLLOWER
I20260924 04:58:59.522109 2015 consensus_queue.cc:260] T 9e9ec8ea5f0244478c9e685bdbd9233e P 3856444a605042b69a7e3159ec5e35b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "239a221e14974ed1be41fa89fe85928f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44377 } } peers { permanent_uuid: "3856444a605042b69a7e3159ec5e35b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40105 } } peers { permanent_uuid: "b0d906902dcd4df68b5f6f3d2de3eb88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37411 } }
I20260924 04:58:59.522219 2015 ts_tablet_manager.cc:1428] T 9e9ec8ea5f0244478c9e685bdbd9233e P 3856444a605042b69a7e3159ec5e35b0: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 04:58:59.522292 2036 raft_consensus.cc:2258] T ccf6a5196af7400d9a245e4dc2a109b3 P b0d906902dcd4df68b5f6f3d2de3eb88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:58:59.522959 2036 ts_tablet_manager.cc:1905] T ccf6a5196af7400d9a245e4dc2a109b3 P b0d906902dcd4df68b5f6f3d2de3eb88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:58:59.524290 2036 ts_tablet_manager.cc:1918] T ccf6a5196af7400d9a245e4dc2a109b3 P b0d906902dcd4df68b5f6f3d2de3eb88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:58:59.524349 2036 log.cc:1196] T ccf6a5196af7400d9a245e4dc2a109b3 P b0d906902dcd4df68b5f6f3d2de3eb88: Deleting WAL directory at /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225929848825-392-0/minicluster-data/ts-2/wal/wals/ccf6a5196af7400d9a245e4dc2a109b3
I20260924 04:58:59.524533 2036 ts_tablet_manager.cc:1939] T ccf6a5196af7400d9a245e4dc2a109b3 P b0d906902dcd4df68b5f6f3d2de3eb88: Deleting consensus metadata
I20260924 04:58:59.524878 1488 catalog_manager.cc:4841] TS b0d906902dcd4df68b5f6f3d2de3eb88 (127.0.98.3:37411): tablet ccf6a5196af7400d9a245e4dc2a109b3 (table test-table [id=671282291b1546d091d22a72fd56de21]) successfully deleted
I20260924 04:58:59.648099 2022 raft_consensus.cc:482] T 9e9ec8ea5f0244478c9e685bdbd9233e P b0d906902dcd4df68b5f6f3d2de3eb88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:58:59.648202 2022 raft_consensus.cc:504] T 9e9ec8ea5f0244478c9e685bdbd9233e P b0d906902dcd4df68b5f6f3d2de3eb88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "239a221e14974ed1be41fa89fe85928f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44377 } } peers { permanent_uuid: "3856444a605042b69a7e3159ec5e35b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40105 } } peers { permanent_uuid: "b0d906902dcd4df68b5f6f3d2de3eb88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37411 } }
I20260924 04:58:59.648363 2022 leader_election.cc:288] T 9e9ec8ea5f0244478c9e685bdbd9233e P b0d906902dcd4df68b5f6f3d2de3eb88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 239a221e14974ed1be41fa89fe85928f (127.0.98.1:44377), 3856444a605042b69a7e3159ec5e35b0 (127.0.98.2:40105)
I20260924 04:58:59.648612 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e9ec8ea5f0244478c9e685bdbd9233e" candidate_uuid: "b0d906902dcd4df68b5f6f3d2de3eb88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "239a221e14974ed1be41fa89fe85928f" is_pre_election: true
I20260924 04:58:59.648618 1803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e9ec8ea5f0244478c9e685bdbd9233e" candidate_uuid: "b0d906902dcd4df68b5f6f3d2de3eb88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3856444a605042b69a7e3159ec5e35b0" is_pre_election: true
I20260924 04:58:59.648699 1662 raft_consensus.cc:2454] T 9e9ec8ea5f0244478c9e685bdbd9233e P 239a221e14974ed1be41fa89fe85928f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0d906902dcd4df68b5f6f3d2de3eb88 in term 0.
I20260924 04:58:59.648711 1803 raft_consensus.cc:2454] T 9e9ec8ea5f0244478c9e685bdbd9233e P 3856444a605042b69a7e3159ec5e35b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0d906902dcd4df68b5f6f3d2de3eb88 in term 0.
I20260924 04:58:59.648852 1875 leader_election.cc:302] T 9e9ec8ea5f0244478c9e685bdbd9233e P b0d906902dcd4df68b5f6f3d2de3eb88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 239a221e14974ed1be41fa89fe85928f, b0d906902dcd4df68b5f6f3d2de3eb88; no voters:
I20260924 04:58:59.648944 2022 raft_consensus.cc:2781] T 9e9ec8ea5f0244478c9e685bdbd9233e P b0d906902dcd4df68b5f6f3d2de3eb88 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:58:59.648983 2022 raft_consensus.cc:482] T 9e9ec8ea5f0244478c9e685bdbd9233e P b0d906902dcd4df68b5f6f3d2de3eb88 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:58:59.648999 2022 raft_consensus.cc:3037] T 9e9ec8ea5f0244478c9e685bdbd9233e P b0d906902dcd4df68b5f6f3d2de3eb88 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:58:59.649636 2022 raft_consensus.cc:504] T 9e9ec8ea5f0244478c9e685bdbd9233e P b0d906902dcd4df68b5f6f3d2de3eb88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "239a221e14974ed1be41fa89fe85928f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44377 } } peers { permanent_uuid: "3856444a605042b69a7e3159ec5e35b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40105 } } peers { permanent_uuid: "b0d906902dcd4df68b5f6f3d2de3eb88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37411 } }
I20260924 04:58:59.649741 2022 leader_election.cc:288] T 9e9ec8ea5f0244478c9e685bdbd9233e P b0d906902dcd4df68b5f6f3d2de3eb88 [CANDIDATE]: Term 1 election: Requested vote from peers 239a221e14974ed1be41fa89fe85928f (127.0.98.1:44377), 3856444a605042b69a7e3159ec5e35b0 (127.0.98.2:40105)
I20260924 04:58:59.649904 1803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e9ec8ea5f0244478c9e685bdbd9233e" candidate_uuid: "b0d906902dcd4df68b5f6f3d2de3eb88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3856444a605042b69a7e3159ec5e35b0"
I20260924 04:58:59.649911 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e9ec8ea5f0244478c9e685bdbd9233e" candidate_uuid: "b0d906902dcd4df68b5f6f3d2de3eb88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "239a221e14974ed1be41fa89fe85928f"
I20260924 04:58:59.649972 1803 raft_consensus.cc:3037] T 9e9ec8ea5f0244478c9e685bdbd9233e P 3856444a605042b69a7e3159ec5e35b0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:58:59.649981 1662 raft_consensus.cc:3037] T 9e9ec8ea5f0244478c9e685bdbd9233e P 239a221e14974ed1be41fa89fe85928f [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:58:59.650573 1662 raft_consensus.cc:2454] T 9e9ec8ea5f0244478c9e685bdbd9233e P 239a221e14974ed1be41fa89fe85928f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0d906902dcd4df68b5f6f3d2de3eb88 in term 1.
I20260924 04:58:59.650749 1875 leader_election.cc:302] T 9e9ec8ea5f0244478c9e685bdbd9233e P b0d906902dcd4df68b5f6f3d2de3eb88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 239a221e14974ed1be41fa89fe85928f, b0d906902dcd4df68b5f6f3d2de3eb88; no voters:
I20260924 04:58:59.650853 2022 raft_consensus.cc:2781] T 9e9ec8ea5f0244478c9e685bdbd9233e P b0d906902dcd4df68b5f6f3d2de3eb88 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:58:59.650906 1803 raft_consensus.cc:2454] T 9e9ec8ea5f0244478c9e685bdbd9233e P 3856444a605042b69a7e3159ec5e35b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0d906902dcd4df68b5f6f3d2de3eb88 in term 1.
I20260924 04:58:59.650923 2022 raft_consensus.cc:686] T 9e9ec8ea5f0244478c9e685bdbd9233e P b0d906902dcd4df68b5f6f3d2de3eb88 [term 1 LEADER]: Becoming Leader. State: Replica: b0d906902dcd4df68b5f6f3d2de3eb88, State: Running, Role: LEADER
I20260924 04:58:59.650987 2022 consensus_queue.cc:237] T 9e9ec8ea5f0244478c9e685bdbd9233e P b0d906902dcd4df68b5f6f3d2de3eb88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "239a221e14974ed1be41fa89fe85928f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44377 } } peers { permanent_uuid: "3856444a605042b69a7e3159ec5e35b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40105 } } peers { permanent_uuid: "b0d906902dcd4df68b5f6f3d2de3eb88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37411 } }
I20260924 04:58:59.651546 1503 catalog_manager.cc:5495] T 9e9ec8ea5f0244478c9e685bdbd9233e P b0d906902dcd4df68b5f6f3d2de3eb88 reported cstate change: term changed from 0 to 1, leader changed from <none> to b0d906902dcd4df68b5f6f3d2de3eb88 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "b0d906902dcd4df68b5f6f3d2de3eb88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "239a221e14974ed1be41fa89fe85928f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3856444a605042b69a7e3159ec5e35b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40105 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b0d906902dcd4df68b5f6f3d2de3eb88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37411 } health_report { overall_health: HEALTHY } } }
W20260924 04:58:59.672449 1991 tablet.cc:2369] T 9e9ec8ea5f0244478c9e685bdbd9233e P b0d906902dcd4df68b5f6f3d2de3eb88: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 04:58:59 dist-test-slave-70st krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225936, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:58:59.692302 1883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:58:59.686150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42078 (local address 127.0.98.62:42079)
0924 04:58:59.686299 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 04:58:59.686302 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:58:59.686382 (+ 80us) server_negotiation.cc:407] Connection header received
0924 04:58:59.686450 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:58:59.686451 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:58:59.686461 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:58:59.686517 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:58:59.686973 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:58:59.687444 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:58:59.688107 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:58:59.688273 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:58:59.688327 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:58:59.690240 (+ 1913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:58:59.690241 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:58:59.690242 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:58:59.690249 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:58:59.691263 (+ 1014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:58:59.691591 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:58:59.691592 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:58:59.691592 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:58:59.691640 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:58:59.691790 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:58:59.691791 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:58:59.691791 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:58:59.691892 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:58:59.691944 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 04:58:59.692084 (+ 140us) server_negotiation.cc:299] Negotiation successful
0924 04:58:59.692179 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":121}
I20260924 04:58:59.692373 2048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:58:59.686073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42079 (local address 127.0.0.1:42078)
0924 04:58:59.686308 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0924 04:58:59.686319 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 04:58:59.686392 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:58:59.686528 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:58:59.686529 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:58:59.686846 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:58:59.686892 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:58:59.686895 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:58:59.687527 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:58:59.687528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:58:59.688038 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:58:59.688040 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:58:59.688330 (+ 290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:58:59.688331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:58:59.688382 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:58:59.688835 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:58:59.688839 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:58:59.690161 (+ 1322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:58:59.691333 (+ 1172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:58:59.691339 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:58:59.691342 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:58:59.691529 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:58:59.691690 (+ 161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:58:59.691691 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:58:59.691691 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:58:59.691743 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:58:59.691941 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:58:59.691948 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:58:59.692000 (+ 52us) client_negotiation.cc:769] Sending connection context
0924 04:58:59.692056 (+ 56us) client_negotiation.cc:240] Negotiation successful
0924 04:58:59.692238 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"mutex_wait_us":86,"thread_start_us":83,"threads_started":1}
I20260924 04:58:59.694162 1503 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42078:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 04:58:59.694346 1503 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:58:59.695573 1642 tablet_service.cc:1467] Processing CreateTablet for tablet 6e24930774754802bb49c79b722da938 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7bfa079b8f5f43da97389c72df128430]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 04:58:59.695641 1924 tablet_service.cc:1467] Processing CreateTablet for tablet 6e24930774754802bb49c79b722da938 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7bfa079b8f5f43da97389c72df128430]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 04:58:59.695636 1783 tablet_service.cc:1467] Processing CreateTablet for tablet 6e24930774754802bb49c79b722da938 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7bfa079b8f5f43da97389c72df128430]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 04:58:59.695726 1642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e24930774754802bb49c79b722da938. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:58:59.695776 1783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e24930774754802bb49c79b722da938. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:58:59.695776 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e24930774754802bb49c79b722da938. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:58:59.696939 2014 tablet_bootstrap.cc:492] T 6e24930774754802bb49c79b722da938 P 239a221e14974ed1be41fa89fe85928f: Bootstrap starting.
I20260924 04:58:59.697180 2018 tablet_bootstrap.cc:492] T 6e24930774754802bb49c79b722da938 P b0d906902dcd4df68b5f6f3d2de3eb88: Bootstrap starting.
I20260924 04:58:59.697369 2014 tablet_bootstrap.cc:654] T 6e24930774754802bb49c79b722da938 P 239a221e14974ed1be41fa89fe85928f: Neither blocks nor log segments found. Creating new log.
I20260924 04:58:59.697492 2018 tablet_bootstrap.cc:654] T 6e24930774754802bb49c79b722da938 P b0d906902dcd4df68b5f6f3d2de3eb88: Neither blocks nor log segments found. Creating new log.
I20260924 04:58:59.697784 2015 tablet_bootstrap.cc:492] T 6e24930774754802bb49c79b722da938 P 3856444a605042b69a7e3159ec5e35b0: Bootstrap starting.
I20260924 04:58:59.698021 2018 tablet_bootstrap.cc:492] T 6e24930774754802bb49c79b722da938 P b0d906902dcd4df68b5f6f3d2de3eb88: No bootstrap required, opened a new log
I20260924 04:58:59.698056 2018 ts_tablet_manager.cc:1397] T 6e24930774754802bb49c79b722da938 P b0d906902dcd4df68b5f6f3d2de3eb88: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:58:59.698145 2018 raft_consensus.cc:348] T 6e24930774754802bb49c79b722da938 P b0d906902dcd4df68b5f6f3d2de3eb88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "239a221e14974ed1be41fa89fe85928f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44377 } } peers { permanent_uuid: "3856444a605042b69a7e3159ec5e35b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40105 } } peers { permanent_uuid: "b0d906902dcd4df68b5f6f3d2de3eb88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37411 } }
I20260924 04:58:59.698195 2018 raft_consensus.cc:374] T 6e24930774754802bb49c79b722da938 P b0d906902dcd4df68b5f6f3d2de3eb88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:58:59.698212 2018 raft_consensus.cc:729] T 6e24930774754802bb49c79b722da938 P b0d906902dcd4df68b5f6f3d2de3eb88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0d906902dcd4df68b5f6f3d2de3eb88, State: Initialized, Role: FOLLOWER
I20260924 04:58:59.698217 2015 tablet_bootstrap.cc:654] T 6e24930774754802bb49c79b722da938 P 3856444a605042b69a7e3159ec5e35b0: Neither blocks nor log segments found. Creating new log.
I20260924 04:58:59.698241 2018 consensus_queue.cc:260] T 6e24930774754802bb49c79b722da938 P b0d906902dcd4df68b5f6f3d2de3eb88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "239a221e14974ed1be41fa89fe85928f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44377 } } peers { permanent_uuid: "3856444a605042b69a7e3159ec5e35b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40105 } } peers { permanent_uuid: "b0d906902dcd4df68b5f6f3d2de3eb88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37411 } }
I20260924 04:58:59.698323 2018 ts_tablet_manager.cc:1428] T 6e24930774754802bb49c79b722da938 P b0d906902dcd4df68b5f6f3d2de3eb88: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 04:58:59.698406 2022 tablet_replica.cc:439] T 6e24930774754802bb49c79b722da938 P b0d906902dcd4df68b5f6f3d2de3eb88: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "239a221e14974ed1be41fa89fe85928f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44377 } } peers { permanent_uuid: "3856444a605042b69a7e3159ec5e35b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40105 } } peers { permanent_uuid: "b0d906902dcd4df68b5f6f3d2de3eb88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37411 } } }
I20260924 04:58:59.698463 2022 tablet_replica.cc:442] T 6e24930774754802bb49c79b722da938 P b0d906902dcd4df68b5f6f3d2de3eb88: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:58:59.698769 2014 tablet_bootstrap.cc:492] T 6e24930774754802bb49c79b722da938 P 239a221e14974ed1be41fa89fe85928f: No bootstrap required, opened a new log
I20260924 04:58:59.698822 2014 ts_tablet_manager.cc:1397] T 6e24930774754802bb49c79b722da938 P 239a221e14974ed1be41fa89fe85928f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 04:58:59.698951 2015 tablet_bootstrap.cc:492] T 6e24930774754802bb49c79b722da938 P 3856444a605042b69a7e3159ec5e35b0: No bootstrap required, opened a new log
I20260924 04:58:59.698949 2014 raft_consensus.cc:348] T 6e24930774754802bb49c79b722da938 P 239a221e14974ed1be41fa89fe85928f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "239a221e14974ed1be41fa89fe85928f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44377 } } peers { permanent_uuid: "3856444a605042b69a7e3159ec5e35b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40105 } } peers { permanent_uuid: "b0d906902dcd4df68b5f6f3d2de3eb88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37411 } }
I20260924 04:58:59.698994 2015 ts_tablet_manager.cc:1397] T 6e24930774754802bb49c79b722da938 P 3856444a605042b69a7e3159ec5e35b0: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:58:59.699012 2014 raft_consensus.cc:374] T 6e24930774754802bb49c79b722da938 P 239a221e14974ed1be41fa89fe85928f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:58:59.699039 2014 raft_consensus.cc:729] T 6e24930774754802bb49c79b722da938 P 239a221e14974ed1be41fa89fe85928f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 239a221e14974ed1be41fa89fe85928f, State: Initialized, Role: FOLLOWER
I20260924 04:58:59.699085 2014 consensus_queue.cc:260] T 6e24930774754802bb49c79b722da938 P 239a221e14974ed1be41fa89fe85928f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "239a221e14974ed1be41fa89fe85928f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44377 } } peers { permanent_uuid: "3856444a605042b69a7e3159ec5e35b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40105 } } peers { permanent_uuid: "b0d906902dcd4df68b5f6f3d2de3eb88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37411 } }
I20260924 04:58:59.699105 2015 raft_consensus.cc:348] T 6e24930774754802bb49c79b722da938 P 3856444a605042b69a7e3159ec5e35b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "239a221e14974ed1be41fa89fe85928f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44377 } } peers { permanent_uuid: "3856444a605042b69a7e3159ec5e35b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40105 } } peers { permanent_uuid: "b0d906902dcd4df68b5f6f3d2de3eb88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37411 } }
I20260924 04:58:59.699167 2014 ts_tablet_manager.cc:1428] T 6e24930774754802bb49c79b722da938 P 239a221e14974ed1be41fa89fe85928f: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 04:58:59.699169 2015 raft_consensus.cc:374] T 6e24930774754802bb49c79b722da938 P 3856444a605042b69a7e3159ec5e35b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:58:59.699208 2015 raft_consensus.cc:729] T 6e24930774754802bb49c79b722da938 P 3856444a605042b69a7e3159ec5e35b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3856444a605042b69a7e3159ec5e35b0, State: Initialized, Role: FOLLOWER
I20260924 04:58:59.699257 2015 consensus_queue.cc:260] T 6e24930774754802bb49c79b722da938 P 3856444a605042b69a7e3159ec5e35b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "239a221e14974ed1be41fa89fe85928f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44377 } } peers { permanent_uuid: "3856444a605042b69a7e3159ec5e35b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40105 } } peers { permanent_uuid: "b0d906902dcd4df68b5f6f3d2de3eb88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37411 } }
I20260924 04:58:59.699339 2015 ts_tablet_manager.cc:1428] T 6e24930774754802bb49c79b722da938 P 3856444a605042b69a7e3159ec5e35b0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 04:58:59.699313 2021 tablet_replica.cc:439] T 6e24930774754802bb49c79b722da938 P 239a221e14974ed1be41fa89fe85928f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "239a221e14974ed1be41fa89fe85928f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44377 } } peers { permanent_uuid: "3856444a605042b69a7e3159ec5e35b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40105 } } peers { permanent_uuid: "b0d906902dcd4df68b5f6f3d2de3eb88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37411 } } }
I20260924 04:58:59.699422 2021 tablet_replica.cc:442] T 6e24930774754802bb49c79b722da938 P 239a221e14974ed1be41fa89fe85928f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:58:59.699417 2020 tablet_replica.cc:439] T 6e24930774754802bb49c79b722da938 P 3856444a605042b69a7e3159ec5e35b0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "239a221e14974ed1be41fa89fe85928f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44377 } } peers { permanent_uuid: "3856444a605042b69a7e3159ec5e35b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40105 } } peers { permanent_uuid: "b0d906902dcd4df68b5f6f3d2de3eb88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37411 } } }
I20260924 04:58:59.699534 2020 tablet_replica.cc:442] T 6e24930774754802bb49c79b722da938 P 3856444a605042b69a7e3159ec5e35b0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:58:59.789377 2020 raft_consensus.cc:482] T 6e24930774754802bb49c79b722da938 P 3856444a605042b69a7e3159ec5e35b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:58:59.789494 2020 raft_consensus.cc:504] T 6e24930774754802bb49c79b722da938 P 3856444a605042b69a7e3159ec5e35b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "239a221e14974ed1be41fa89fe85928f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44377 } } peers { permanent_uuid: "3856444a605042b69a7e3159ec5e35b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40105 } } peers { permanent_uuid: "b0d906902dcd4df68b5f6f3d2de3eb88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37411 } }
I20260924 04:58:59.789775 2020 leader_election.cc:288] T 6e24930774754802bb49c79b722da938 P 3856444a605042b69a7e3159ec5e35b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 239a221e14974ed1be41fa89fe85928f (127.0.98.1:44377), b0d906902dcd4df68b5f6f3d2de3eb88 (127.0.98.3:37411)
I20260924 04:58:59.793692 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:58:59.790121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59031 (local address 127.0.98.3:37411)
0924 04:58:59.790179 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 04:58:59.790181 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:58:59.790327 (+ 146us) server_negotiation.cc:407] Connection header received
0924 04:58:59.790506 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:58:59.790507 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:58:59.790517 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:58:59.790561 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:58:59.790667 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:58:59.791380 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:58:59.792815 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:58:59.793349 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:58:59.793436 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:58:59.793448 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0924 04:58:59.793576 (+ 128us) server_negotiation.cc:299] Negotiation successful
0924 04:58:59.793620 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 04:58:59.793720 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:58:59.790159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58847 (local address 127.0.98.1:44377)
0924 04:58:59.790204 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 04:58:59.790206 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:58:59.790307 (+ 101us) server_negotiation.cc:407] Connection header received
0924 04:58:59.790430 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:58:59.790431 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:58:59.790441 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:58:59.790491 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:58:59.790686 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:58:59.791370 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:58:59.792843 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:58:59.793345 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:58:59.793384 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:58:59.793393 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 04:58:59.793595 (+ 202us) server_negotiation.cc:299] Negotiation successful
0924 04:58:59.793637 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 04:58:59.793740 2053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:58:59.789935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37411 (local address 127.0.98.2:59031)
0924 04:58:59.790204 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0924 04:58:59.790240 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 04:58:59.790355 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:58:59.790576 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:58:59.790577 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:58:59.790578 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:58:59.790607 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:58:59.790609 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:58:59.791512 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:58:59.791513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:58:59.792729 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:58:59.792732 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:58:59.793441 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:58:59.793442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:58:59.793515 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:58:59.793519 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 04:58:59.793566 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 04:58:59.793613 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"mutex_wait_us":27,"thread_start_us":124,"threads_started":1}
I20260924 04:58:59.793742 2052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:58:59.789935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44377 (local address 127.0.98.2:58847)
0924 04:58:59.790240 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0924 04:58:59.790251 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 04:58:59.790339 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:58:59.790524 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:58:59.790525 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:58:59.790526 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:58:59.790610 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:58:59.790612 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:58:59.791454 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:58:59.791455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:58:59.792735 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:58:59.792739 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:58:59.793480 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:58:59.793481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:58:59.793534 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:58:59.793537 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 04:58:59.793578 (+ 41us) client_negotiation.cc:240] Negotiation successful
0924 04:58:59.793627 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"mutex_wait_us":55,"thread_start_us":111,"threads_started":1}
I20260924 04:58:59.794013 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e24930774754802bb49c79b722da938" candidate_uuid: "3856444a605042b69a7e3159ec5e35b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "239a221e14974ed1be41fa89fe85928f" is_pre_election: true
I20260924 04:58:59.794114 1662 raft_consensus.cc:2454] T 6e24930774754802bb49c79b722da938 P 239a221e14974ed1be41fa89fe85928f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3856444a605042b69a7e3159ec5e35b0 in term 0.
I20260924 04:58:59.794090 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e24930774754802bb49c79b722da938" candidate_uuid: "3856444a605042b69a7e3159ec5e35b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0d906902dcd4df68b5f6f3d2de3eb88" is_pre_election: true
I20260924 04:58:59.794220 1944 raft_consensus.cc:2454] T 6e24930774754802bb49c79b722da938 P b0d906902dcd4df68b5f6f3d2de3eb88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3856444a605042b69a7e3159ec5e35b0 in term 0.
I20260924 04:58:59.794291 1734 leader_election.cc:302] T 6e24930774754802bb49c79b722da938 P 3856444a605042b69a7e3159ec5e35b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 239a221e14974ed1be41fa89fe85928f, 3856444a605042b69a7e3159ec5e35b0; no voters:
I20260924 04:58:59.794413 2020 raft_consensus.cc:2781] T 6e24930774754802bb49c79b722da938 P 3856444a605042b69a7e3159ec5e35b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:58:59.794452 2020 raft_consensus.cc:482] T 6e24930774754802bb49c79b722da938 P 3856444a605042b69a7e3159ec5e35b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:58:59.794478 2020 raft_consensus.cc:3037] T 6e24930774754802bb49c79b722da938 P 3856444a605042b69a7e3159ec5e35b0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:58:59.795149 2020 raft_consensus.cc:504] T 6e24930774754802bb49c79b722da938 P 3856444a605042b69a7e3159ec5e35b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "239a221e14974ed1be41fa89fe85928f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44377 } } peers { permanent_uuid: "3856444a605042b69a7e3159ec5e35b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40105 } } peers { permanent_uuid: "b0d906902dcd4df68b5f6f3d2de3eb88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37411 } }
I20260924 04:58:59.795267 2020 leader_election.cc:288] T 6e24930774754802bb49c79b722da938 P 3856444a605042b69a7e3159ec5e35b0 [CANDIDATE]: Term 1 election: Requested vote from peers 239a221e14974ed1be41fa89fe85928f (127.0.98.1:44377), b0d906902dcd4df68b5f6f3d2de3eb88 (127.0.98.3:37411)
I20260924 04:58:59.795398 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e24930774754802bb49c79b722da938" candidate_uuid: "3856444a605042b69a7e3159ec5e35b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0d906902dcd4df68b5f6f3d2de3eb88"
I20260924 04:58:59.795419 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e24930774754802bb49c79b722da938" candidate_uuid: "3856444a605042b69a7e3159ec5e35b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "239a221e14974ed1be41fa89fe85928f"
I20260924 04:58:59.795476 1662 raft_consensus.cc:3037] T 6e24930774754802bb49c79b722da938 P 239a221e14974ed1be41fa89fe85928f [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:58:59.795482 1944 raft_consensus.cc:3037] T 6e24930774754802bb49c79b722da938 P b0d906902dcd4df68b5f6f3d2de3eb88 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:58:59.796255 1662 raft_consensus.cc:2454] T 6e24930774754802bb49c79b722da938 P 239a221e14974ed1be41fa89fe85928f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3856444a605042b69a7e3159ec5e35b0 in term 1.
I20260924 04:58:59.796255 1944 raft_consensus.cc:2454] T 6e24930774754802bb49c79b722da938 P b0d906902dcd4df68b5f6f3d2de3eb88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3856444a605042b69a7e3159ec5e35b0 in term 1.
I20260924 04:58:59.796420 1733 leader_election.cc:302] T 6e24930774754802bb49c79b722da938 P 3856444a605042b69a7e3159ec5e35b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3856444a605042b69a7e3159ec5e35b0, b0d906902dcd4df68b5f6f3d2de3eb88; no voters:
I20260924 04:58:59.796515 2020 raft_consensus.cc:2781] T 6e24930774754802bb49c79b722da938 P 3856444a605042b69a7e3159ec5e35b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:58:59.796659 2020 raft_consensus.cc:686] T 6e24930774754802bb49c79b722da938 P 3856444a605042b69a7e3159ec5e35b0 [term 1 LEADER]: Becoming Leader. State: Replica: 3856444a605042b69a7e3159ec5e35b0, State: Running, Role: LEADER
I20260924 04:58:59.796720 2020 consensus_queue.cc:237] T 6e24930774754802bb49c79b722da938 P 3856444a605042b69a7e3159ec5e35b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "239a221e14974ed1be41fa89fe85928f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44377 } } peers { permanent_uuid: "3856444a605042b69a7e3159ec5e35b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40105 } } peers { permanent_uuid: "b0d906902dcd4df68b5f6f3d2de3eb88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37411 } }
I20260924 04:58:59.797035 2054 tablet_replica.cc:439] T 6e24930774754802bb49c79b722da938 P 3856444a605042b69a7e3159ec5e35b0: TxnStatusTablet state changed. Reason: New leader 3856444a605042b69a7e3159ec5e35b0. Latest consensus state: current_term: 1 leader_uuid: "3856444a605042b69a7e3159ec5e35b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "239a221e14974ed1be41fa89fe85928f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44377 } } peers { permanent_uuid: "3856444a605042b69a7e3159ec5e35b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40105 } } peers { permanent_uuid: "b0d906902dcd4df68b5f6f3d2de3eb88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37411 } } }
I20260924 04:58:59.797127 2054 tablet_replica.cc:442] T 6e24930774754802bb49c79b722da938 P 3856444a605042b69a7e3159ec5e35b0: This TxnStatusTablet replica's current role is: LEADER
I20260924 04:58:59.797494 2056 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 04:58:59.797451 1502 catalog_manager.cc:5495] T 6e24930774754802bb49c79b722da938 P 3856444a605042b69a7e3159ec5e35b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3856444a605042b69a7e3159ec5e35b0 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "3856444a605042b69a7e3159ec5e35b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "239a221e14974ed1be41fa89fe85928f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3856444a605042b69a7e3159ec5e35b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40105 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b0d906902dcd4df68b5f6f3d2de3eb88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37411 } health_report { overall_health: UNKNOWN } } }
I20260924 04:58:59.797601 2056 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 04:58:59.815393 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:58:59.812985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37692 (local address 127.0.98.2:40105)
0924 04:58:59.813041 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 04:58:59.813043 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:58:59.813051 (+ 8us) server_negotiation.cc:407] Connection header received
0924 04:58:59.813138 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:58:59.813139 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:58:59.813158 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:58:59.813210 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:58:59.813328 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:58:59.813831 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:58:59.814665 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:58:59.814851 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:58:59.814927 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:58:59.815071 (+ 144us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:58:59.815168 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:58:59.815228 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 04:58:59.815290 (+ 62us) server_negotiation.cc:299] Negotiation successful
0924 04:58:59.815311 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 04:58:59.815404 2048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:58:59.812912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40105 (local address 127.0.0.1:37692)
0924 04:58:59.813011 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0924 04:58:59.813023 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:58:59.813070 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:58:59.813219 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:58:59.813220 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:58:59.813221 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 04:58:59.813271 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:58:59.813274 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:58:59.813909 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:58:59.813910 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:58:59.814601 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:58:59.814603 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:58:59.814913 (+ 310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:58:59.814914 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:58:59.814967 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:58:59.814995 (+ 28us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 04:58:59.815222 (+ 227us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 04:58:59.815229 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 04:58:59.815284 (+ 55us) client_negotiation.cc:240] Negotiation successful
0924 04:58:59.815332 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260924 04:58:59.816481 1662 raft_consensus.cc:1261] T 6e24930774754802bb49c79b722da938 P 239a221e14974ed1be41fa89fe85928f [term 1 FOLLOWER]: Refusing update from remote peer 3856444a605042b69a7e3159ec5e35b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:58:59.816612 1944 raft_consensus.cc:1261] T 6e24930774754802bb49c79b722da938 P b0d906902dcd4df68b5f6f3d2de3eb88 [term 1 FOLLOWER]: Refusing update from remote peer 3856444a605042b69a7e3159ec5e35b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:58:59.816720 2020 consensus_queue.cc:1035] T 6e24930774754802bb49c79b722da938 P 3856444a605042b69a7e3159ec5e35b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "239a221e14974ed1be41fa89fe85928f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:58:59.816881 2020 consensus_queue.cc:1035] T 6e24930774754802bb49c79b722da938 P 3856444a605042b69a7e3159ec5e35b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0d906902dcd4df68b5f6f3d2de3eb88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:58:59.817443 2022 tablet_replica.cc:439] T 6e24930774754802bb49c79b722da938 P b0d906902dcd4df68b5f6f3d2de3eb88: TxnStatusTablet state changed. Reason: New leader 3856444a605042b69a7e3159ec5e35b0. Latest consensus state: current_term: 1 leader_uuid: "3856444a605042b69a7e3159ec5e35b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "239a221e14974ed1be41fa89fe85928f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44377 } } peers { permanent_uuid: "3856444a605042b69a7e3159ec5e35b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40105 } } peers { permanent_uuid: "b0d906902dcd4df68b5f6f3d2de3eb88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37411 } } }
I20260924 04:58:59.817541 2022 tablet_replica.cc:442] T 6e24930774754802bb49c79b722da938 P b0d906902dcd4df68b5f6f3d2de3eb88: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:58:59.817986 2021 tablet_replica.cc:439] T 6e24930774754802bb49c79b722da938 P 239a221e14974ed1be41fa89fe85928f: TxnStatusTablet state changed. Reason: New leader 3856444a605042b69a7e3159ec5e35b0. Latest consensus state: current_term: 1 leader_uuid: "3856444a605042b69a7e3159ec5e35b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "239a221e14974ed1be41fa89fe85928f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44377 } } peers { permanent_uuid: "3856444a605042b69a7e3159ec5e35b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40105 } } peers { permanent_uuid: "b0d906902dcd4df68b5f6f3d2de3eb88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37411 } } }
I20260924 04:58:59.818074 2021 tablet_replica.cc:442] T 6e24930774754802bb49c79b722da938 P 239a221e14974ed1be41fa89fe85928f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:58:59.818033 2059 tablet_replica.cc:439] T 6e24930774754802bb49c79b722da938 P 3856444a605042b69a7e3159ec5e35b0: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3856444a605042b69a7e3159ec5e35b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "239a221e14974ed1be41fa89fe85928f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44377 } } peers { permanent_uuid: "3856444a605042b69a7e3159ec5e35b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40105 } } peers { permanent_uuid: "b0d906902dcd4df68b5f6f3d2de3eb88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37411 } } }
I20260924 04:58:59.818120 2059 tablet_replica.cc:442] T 6e24930774754802bb49c79b722da938 P 3856444a605042b69a7e3159ec5e35b0: This TxnStatusTablet replica's current role is: LEADER
I20260924 04:58:59.818692 2020 tablet_replica.cc:439] T 6e24930774754802bb49c79b722da938 P 3856444a605042b69a7e3159ec5e35b0: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3856444a605042b69a7e3159ec5e35b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "239a221e14974ed1be41fa89fe85928f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44377 } } peers { permanent_uuid: "3856444a605042b69a7e3159ec5e35b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40105 } } peers { permanent_uuid: "b0d906902dcd4df68b5f6f3d2de3eb88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37411 } } }
I20260924 04:58:59.818761 2020 tablet_replica.cc:442] T 6e24930774754802bb49c79b722da938 P 3856444a605042b69a7e3159ec5e35b0: This TxnStatusTablet replica's current role is: LEADER
I20260924 04:58:59.819427 2022 tablet_replica.cc:439] T 6e24930774754802bb49c79b722da938 P b0d906902dcd4df68b5f6f3d2de3eb88: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3856444a605042b69a7e3159ec5e35b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "239a221e14974ed1be41fa89fe85928f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44377 } } peers { permanent_uuid: "3856444a605042b69a7e3159ec5e35b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40105 } } peers { permanent_uuid: "b0d906902dcd4df68b5f6f3d2de3eb88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37411 } } }
I20260924 04:58:59.819514 2022 tablet_replica.cc:442] T 6e24930774754802bb49c79b722da938 P b0d906902dcd4df68b5f6f3d2de3eb88: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:58:59.820188 2030 mvcc.cc:205] Tried to move back new op lower bound from 7332765449486712832 to 7332765449407967232. Current Snapshot: MvccSnapshot[applied={T|T < 7332765449486712832 or (T in {7332765449486712832})}]
I20260924 04:58:59.820392 2021 tablet_replica.cc:439] T 6e24930774754802bb49c79b722da938 P 239a221e14974ed1be41fa89fe85928f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3856444a605042b69a7e3159ec5e35b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "239a221e14974ed1be41fa89fe85928f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44377 } } peers { permanent_uuid: "3856444a605042b69a7e3159ec5e35b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40105 } } peers { permanent_uuid: "b0d906902dcd4df68b5f6f3d2de3eb88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37411 } } }
I20260924 04:58:59.820470 2021 tablet_replica.cc:442] T 6e24930774754802bb49c79b722da938 P 239a221e14974ed1be41fa89fe85928f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:58:59.824062 1996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:58:59.821435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40105 (local address 127.0.98.3:37551)
0924 04:58:59.821507 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0924 04:58:59.821518 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 04:58:59.821556 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:58:59.821705 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:58:59.821706 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:58:59.821708 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 04:58:59.821753 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:58:59.821755 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:58:59.822616 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:58:59.822617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:58:59.823291 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:58:59.823293 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:58:59.823732 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:58:59.823733 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:58:59.823773 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:58:59.823787 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 04:58:59.823942 (+ 155us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 04:58:59.823943 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 04:58:59.823971 (+ 28us) client_negotiation.cc:240] Negotiation successful
0924 04:58:59.824010 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260924 04:58:59.824079 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:58:59.821497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37551 (local address 127.0.98.2:40105)
0924 04:58:59.821549 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 04:58:59.821551 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:58:59.821560 (+ 9us) server_negotiation.cc:407] Connection header received
0924 04:58:59.821600 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:58:59.821601 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:58:59.821622 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:58:59.821696 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:58:59.821812 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:58:59.822543 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:58:59.823359 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:58:59.823664 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:58:59.823734 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:58:59.823858 (+ 124us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:58:59.823891 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:58:59.823938 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 04:58:59.823993 (+ 55us) server_negotiation.cc:299] Negotiation successful
0924 04:58:59.824010 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 04:58:59.825884 1803 raft_consensus.cc:1261] T 9e9ec8ea5f0244478c9e685bdbd9233e P 3856444a605042b69a7e3159ec5e35b0 [term 1 FOLLOWER]: Refusing update from remote peer b0d906902dcd4df68b5f6f3d2de3eb88: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:58:59.825903 1662 raft_consensus.cc:1261] T 9e9ec8ea5f0244478c9e685bdbd9233e P 239a221e14974ed1be41fa89fe85928f [term 1 FOLLOWER]: Refusing update from remote peer b0d906902dcd4df68b5f6f3d2de3eb88: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:58:59.826102 2022 consensus_queue.cc:1035] T 9e9ec8ea5f0244478c9e685bdbd9233e P b0d906902dcd4df68b5f6f3d2de3eb88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "239a221e14974ed1be41fa89fe85928f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:58:59.826187 2024 consensus_queue.cc:1035] T 9e9ec8ea5f0244478c9e685bdbd9233e P b0d906902dcd4df68b5f6f3d2de3eb88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3856444a605042b69a7e3159ec5e35b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:58:59.834312 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:58:59.831739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59735 (local address 127.0.98.3:37411)
0924 04:58:59.831778 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 04:58:59.831780 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:58:59.831906 (+ 126us) server_negotiation.cc:407] Connection header received
0924 04:58:59.831970 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:58:59.831971 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:58:59.831981 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:58:59.832016 (+ 35us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:58:59.832151 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:58:59.832608 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:58:59.833719 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:58:59.833915 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:58:59.833963 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:58:59.834133 (+ 170us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:58:59.834163 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:58:59.834192 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0924 04:58:59.834247 (+ 55us) server_negotiation.cc:299] Negotiation successful
0924 04:58:59.834262 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 04:58:59.834425 2067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:58:59.831654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37411 (local address 127.0.98.2:59735)
0924 04:58:59.831847 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0924 04:58:59.831858 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 04:58:59.831917 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:58:59.832033 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:58:59.832034 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:58:59.832035 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 04:58:59.832100 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:58:59.832102 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:58:59.832675 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:58:59.832675 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:58:59.833654 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:58:59.833657 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:58:59.833987 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:58:59.833988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:58:59.834053 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:58:59.834067 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 04:58:59.834199 (+ 132us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 04:58:59.834200 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 04:58:59.834245 (+ 45us) client_negotiation.cc:240] Negotiation successful
0924 04:58:59.834284 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":69,"threads_started":1}
I20260924 04:58:59.843149 1504 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42072:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:58:59.843240 1504 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42072:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:58:59.843808 1504 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d11579be4182427e90eaae6d4647dff1: Sending DeleteTablet for 3 replicas of tablet 9e9ec8ea5f0244478c9e685bdbd9233e
I20260924 04:58:59.844028 1924 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e9ec8ea5f0244478c9e685bdbd9233e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:58:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51944
I20260924 04:58:59.844117 1783 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e9ec8ea5f0244478c9e685bdbd9233e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:58:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37684
I20260924 04:58:59.844139 1642 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e9ec8ea5f0244478c9e685bdbd9233e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:58:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40854
I20260924 04:58:59.844203 2038 tablet_replica.cc:330] T 9e9ec8ea5f0244478c9e685bdbd9233e P 3856444a605042b69a7e3159ec5e35b0: stopping tablet replica
I20260924 04:58:59.844216 2036 tablet_replica.cc:330] T 9e9ec8ea5f0244478c9e685bdbd9233e P b0d906902dcd4df68b5f6f3d2de3eb88: stopping tablet replica
I20260924 04:58:59.844246 2038 raft_consensus.cc:2229] T 9e9ec8ea5f0244478c9e685bdbd9233e P 3856444a605042b69a7e3159ec5e35b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:58:59.844266 2036 raft_consensus.cc:2229] T 9e9ec8ea5f0244478c9e685bdbd9233e P b0d906902dcd4df68b5f6f3d2de3eb88 [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:58:59.844277 2038 raft_consensus.cc:2258] T 9e9ec8ea5f0244478c9e685bdbd9233e P 3856444a605042b69a7e3159ec5e35b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:58:59.844310 2036 raft_consensus.cc:2258] T 9e9ec8ea5f0244478c9e685bdbd9233e P b0d906902dcd4df68b5f6f3d2de3eb88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:58:59.844370 2037 tablet_replica.cc:330] T 9e9ec8ea5f0244478c9e685bdbd9233e P 239a221e14974ed1be41fa89fe85928f: stopping tablet replica
I20260924 04:58:59.844409 2037 raft_consensus.cc:2229] T 9e9ec8ea5f0244478c9e685bdbd9233e P 239a221e14974ed1be41fa89fe85928f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:58:59.844448 2037 raft_consensus.cc:2258] T 9e9ec8ea5f0244478c9e685bdbd9233e P 239a221e14974ed1be41fa89fe85928f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:58:59.844579 2038 ts_tablet_manager.cc:1905] T 9e9ec8ea5f0244478c9e685bdbd9233e P 3856444a605042b69a7e3159ec5e35b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:58:59.844847 2036 ts_tablet_manager.cc:1905] T 9e9ec8ea5f0244478c9e685bdbd9233e P b0d906902dcd4df68b5f6f3d2de3eb88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:58:59.846114 2037 ts_tablet_manager.cc:1905] T 9e9ec8ea5f0244478c9e685bdbd9233e P 239a221e14974ed1be41fa89fe85928f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:58:59.846323 2036 ts_tablet_manager.cc:1918] T 9e9ec8ea5f0244478c9e685bdbd9233e P b0d906902dcd4df68b5f6f3d2de3eb88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 04:58:59.846333 2038 ts_tablet_manager.cc:1918] T 9e9ec8ea5f0244478c9e685bdbd9233e P 3856444a605042b69a7e3159ec5e35b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 04:58:59.846374 2036 log.cc:1196] T 9e9ec8ea5f0244478c9e685bdbd9233e P b0d906902dcd4df68b5f6f3d2de3eb88: Deleting WAL directory at /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225929848825-392-0/minicluster-data/ts-2/wal/wals/9e9ec8ea5f0244478c9e685bdbd9233e
I20260924 04:58:59.846383 2038 log.cc:1196] T 9e9ec8ea5f0244478c9e685bdbd9233e P 3856444a605042b69a7e3159ec5e35b0: Deleting WAL directory at /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225929848825-392-0/minicluster-data/ts-1/wal/wals/9e9ec8ea5f0244478c9e685bdbd9233e
I20260924 04:58:59.846554 2036 ts_tablet_manager.cc:1939] T 9e9ec8ea5f0244478c9e685bdbd9233e P b0d906902dcd4df68b5f6f3d2de3eb88: Deleting consensus metadata
I20260924 04:58:59.846596 2038 ts_tablet_manager.cc:1939] T 9e9ec8ea5f0244478c9e685bdbd9233e P 3856444a605042b69a7e3159ec5e35b0: Deleting consensus metadata
I20260924 04:58:59.846823 1488 catalog_manager.cc:4841] TS b0d906902dcd4df68b5f6f3d2de3eb88 (127.0.98.3:37411): tablet 9e9ec8ea5f0244478c9e685bdbd9233e (table test-table [id=e755be6f8db147cd9cc4ac0ffd989ba7]) successfully deleted
I20260924 04:58:59.846892 1491 catalog_manager.cc:4841] TS 3856444a605042b69a7e3159ec5e35b0 (127.0.98.2:40105): tablet 9e9ec8ea5f0244478c9e685bdbd9233e (table test-table [id=e755be6f8db147cd9cc4ac0ffd989ba7]) successfully deleted
I20260924 04:58:59.847565 2037 ts_tablet_manager.cc:1918] T 9e9ec8ea5f0244478c9e685bdbd9233e P 239a221e14974ed1be41fa89fe85928f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 04:58:59.847613 2037 log.cc:1196] T 9e9ec8ea5f0244478c9e685bdbd9233e P 239a221e14974ed1be41fa89fe85928f: Deleting WAL directory at /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790225929848825-392-0/minicluster-data/ts-0/wal/wals/9e9ec8ea5f0244478c9e685bdbd9233e
I20260924 04:58:59.847770 2037 ts_tablet_manager.cc:1939] T 9e9ec8ea5f0244478c9e685bdbd9233e P 239a221e14974ed1be41fa89fe85928f: Deleting consensus metadata
I20260924 04:58:59.848095 1489 catalog_manager.cc:4841] TS 239a221e14974ed1be41fa89fe85928f (127.0.98.1:44377): tablet 9e9ec8ea5f0244478c9e685bdbd9233e (table test-table [id=e755be6f8db147cd9cc4ac0ffd989ba7]) successfully deleted
Sep 24 04:58:59 dist-test-slave-70st krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225939, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 04:58:59.858121 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:58:59.849691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40860 (local address 127.0.98.1:44377)
0924 04:58:59.849732 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 04:58:59.849734 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:58:59.849875 (+ 141us) server_negotiation.cc:407] Connection header received
0924 04:58:59.849916 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:58:59.849917 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:58:59.849930 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:58:59.849994 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:58:59.850487 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:58:59.851241 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:58:59.851959 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:58:59.852268 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:58:59.852329 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:58:59.855197 (+ 2868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:58:59.855213 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:58:59.855213 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:58:59.855221 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:58:59.856968 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:58:59.857289 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:58:59.857294 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:58:59.857294 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:58:59.857358 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:58:59.857547 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:58:59.857548 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:58:59.857548 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:58:59.857727 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:58:59.857889 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0924 04:58:59.857925 (+ 36us) server_negotiation.cc:299] Negotiation successful
0924 04:58:59.858011 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
W20260924 04:58:59.858378 1702 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40860
I20260924 04:58:59.865444 1883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:58:59.859281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42088 (local address 127.0.98.62:42079)
0924 04:58:59.859325 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 04:58:59.859327 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:58:59.859356 (+ 29us) server_negotiation.cc:407] Connection header received
0924 04:58:59.859418 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:58:59.859419 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:58:59.859427 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:58:59.859468 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:58:59.860073 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:58:59.860634 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:58:59.861655 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:58:59.861863 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:58:59.861915 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:58:59.863210 (+ 1295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:58:59.863211 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:58:59.863212 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:58:59.863218 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:58:59.864460 (+ 1242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:58:59.864804 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:58:59.864805 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:58:59.864805 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:58:59.864855 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:58:59.865035 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:58:59.865036 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:58:59.865036 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:58:59.865128 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:58:59.865195 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 04:58:59.865303 (+ 108us) server_negotiation.cc:299] Negotiation successful
0924 04:58:59.865358 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
W20260924 04:58:59.865688 1504 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42088
I20260924 04:58:59.866271 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 1576
I20260924 04:58:59.872107 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 1717
I20260924 04:58:59.876519 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 1858
I20260924 04:58:59.881110 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 1472
2026-09-24T04:58:59Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (6479 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225929848825-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:58:59 dist-test-slave-70st krb5kdc[2076](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:58:59 dist-test-slave-70st krb5kdc[2076](info): set up 2 sockets
Sep 24 04:58:59 dist-test-slave-70st krb5kdc[2076](info): commencing operation
krb5kdc: starting...
W20260924 04:59:00.824837 392 mini_kdc.cc:121] Time spent starting KDC: real 0.925s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225929848825-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225929848825-392-0/krb5kdc/test-user.keytab.
Sep 24 04:59:00 dist-test-slave-70st krb5kdc[2076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225940, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:59:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:59:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:59:00.929639 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225929848825-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225929848825-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225929848825-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42637
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42637
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:59:01.006031 2089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:59:01.006235 2089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:59:01.006266 2089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:59:01.007462 2089 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:59:01.007505 2089 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 04:59:01.007522 2089 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:59:01.007539 2089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:59:01.007553 2089 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:59:01.007570 2089 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:59:01.009003 2089 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225929848825-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225929848825-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42637
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42637
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2089
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225929848825-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 04:59:01.009227 2089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:59:01.009431 2089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:59:01.013904 2095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:59:01.013952 2094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:59:01.013945 2097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:59:01.014175 2089 server_base.cc:1034] running on GCE node
I20260924 04:59:01.014396 2089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:59:01.014662 2089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:59:01.015813 2089 hybrid_clock.cc:648] HybridClock initialized: now 1790225941015802 us; error 27 us; skew 500 ppm
Sep 24 04:59:01 dist-test-slave-70st krb5kdc[2076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225941, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:59:01.018289 2089 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:59:01.018767 2089 webserver.cc:460] Webserver started at http://127.0.98.62:37177/ using document root <none> and password file <none>
I20260924 04:59:01.018996 2089 fs_manager.cc:359] Metadata directory not provided
I20260924 04:59:01.019042 2089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:59:01.019168 2089 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:59:01.020175 2089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225929848825-392-0/minicluster-data/master-0/data/instance:
uuid: "cb927924e0fe40f185c2f20be51a64f0"
format_stamp: "Formatted at 2026-09-24 04:59:01 on dist-test-slave-70st"
server_key: "4b4c85d4e191834be3dacc52cfba8a9c"
server_key_iv: "fdd9f5707774d9ccdc3ffd4a1efd215f"
server_key_version: "encryptionkey@0"
I20260924 04:59:01.020503 2089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225929848825-392-0/minicluster-data/master-0/wal/instance:
uuid: "cb927924e0fe40f185c2f20be51a64f0"
format_stamp: "Formatted at 2026-09-24 04:59:01 on dist-test-slave-70st"
server_key: "4b4c85d4e191834be3dacc52cfba8a9c"
server_key_iv: "fdd9f5707774d9ccdc3ffd4a1efd215f"
server_key_version: "encryptionkey@0"
I20260924 04:59:01.021867 2089 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 04:59:01.022673 2104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:01.022851 2089 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 04:59:01.022938 2089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225929848825-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225929848825-392-0/minicluster-data/master-0/wal
uuid: "cb927924e0fe40f185c2f20be51a64f0"
format_stamp: "Formatted at 2026-09-24 04:59:01 on dist-test-slave-70st"
server_key: "4b4c85d4e191834be3dacc52cfba8a9c"
server_key_iv: "fdd9f5707774d9ccdc3ffd4a1efd215f"
server_key_version: "encryptionkey@0"
I20260924 04:59:01.023010 2089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225929848825-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225929848825-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225929848825-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:59:01.052714 2089 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:59:01.054764 2089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:59:01.054876 2089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:59:01.059132 2089 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42637
I20260924 04:59:01.059181 2166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42637 every 8 connection(s)
I20260924 04:59:01.059499 2089 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:59:01.060114 2167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:59:01.062342 2167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb927924e0fe40f185c2f20be51a64f0: Bootstrap starting.
I20260924 04:59:01.062901 2167 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cb927924e0fe40f185c2f20be51a64f0: Neither blocks nor log segments found. Creating new log.
I20260924 04:59:01.063158 2167 log.cc:824] T 00000000000000000000000000000000 P cb927924e0fe40f185c2f20be51a64f0: Log is configured to *not* fsync() on all Append() calls
I20260924 04:59:01.063730 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 2089
I20260924 04:59:01.063810 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225929848825-392-0/minicluster-data/master-0/wal/instance
I20260924 04:59:01.063817 2167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb927924e0fe40f185c2f20be51a64f0: No bootstrap required, opened a new log
I20260924 04:59:01.063961 392 external_mini_cluster.cc:1428] Setting key 6166affecbbba961c9f0e678e590a0b6
I20260924 04:59:01.065994 2167 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb927924e0fe40f185c2f20be51a64f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb927924e0fe40f185c2f20be51a64f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42637 } }
I20260924 04:59:01.066128 2167 raft_consensus.cc:374] T 00000000000000000000000000000000 P cb927924e0fe40f185c2f20be51a64f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:59:01.066159 2167 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb927924e0fe40f185c2f20be51a64f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb927924e0fe40f185c2f20be51a64f0, State: Initialized, Role: FOLLOWER
I20260924 04:59:01.066300 2167 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb927924e0fe40f185c2f20be51a64f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb927924e0fe40f185c2f20be51a64f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42637 } }
I20260924 04:59:01.066371 2167 raft_consensus.cc:388] T 00000000000000000000000000000000 P cb927924e0fe40f185c2f20be51a64f0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:59:01.066401 2167 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb927924e0fe40f185c2f20be51a64f0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:59:01.066445 2167 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb927924e0fe40f185c2f20be51a64f0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:59:01.066942 2167 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb927924e0fe40f185c2f20be51a64f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb927924e0fe40f185c2f20be51a64f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42637 } }
I20260924 04:59:01.067034 2167 leader_election.cc:302] T 00000000000000000000000000000000 P cb927924e0fe40f185c2f20be51a64f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cb927924e0fe40f185c2f20be51a64f0; no voters:
I20260924 04:59:01.067186 2167 leader_election.cc:288] T 00000000000000000000000000000000 P cb927924e0fe40f185c2f20be51a64f0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:59:01.067219 2172 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cb927924e0fe40f185c2f20be51a64f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:59:01.067320 2172 raft_consensus.cc:686] T 00000000000000000000000000000000 P cb927924e0fe40f185c2f20be51a64f0 [term 1 LEADER]: Becoming Leader. State: Replica: cb927924e0fe40f185c2f20be51a64f0, State: Running, Role: LEADER
I20260924 04:59:01.067427 2167 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb927924e0fe40f185c2f20be51a64f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:59:01.067510 2172 consensus_queue.cc:237] T 00000000000000000000000000000000 P cb927924e0fe40f185c2f20be51a64f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb927924e0fe40f185c2f20be51a64f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42637 } }
I20260924 04:59:01.067844 2172 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb927924e0fe40f185c2f20be51a64f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb927924e0fe40f185c2f20be51a64f0. Latest consensus state: current_term: 1 leader_uuid: "cb927924e0fe40f185c2f20be51a64f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb927924e0fe40f185c2f20be51a64f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42637 } } }
I20260924 04:59:01.067857 2174 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb927924e0fe40f185c2f20be51a64f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cb927924e0fe40f185c2f20be51a64f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb927924e0fe40f185c2f20be51a64f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42637 } } }
I20260924 04:59:01.068015 2172 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb927924e0fe40f185c2f20be51a64f0 [sys.catalog]: This master's current role is: LEADER
I20260924 04:59:01.068028 2174 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb927924e0fe40f185c2f20be51a64f0 [sys.catalog]: This master's current role is: LEADER
I20260924 04:59:01.068225 2181 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:59:01.068776 2181 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:59:01.071308 2181 catalog_manager.cc:1347] Generated new cluster ID: be264128ed7a43879c2717cea38451bf
I20260924 04:59:01.071362 2181 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 04:59:01 dist-test-slave-70st krb5kdc[2076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225940, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:59:01.075151 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:01.064429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59622 (local address 127.0.98.62:42637)
0924 04:59:01.064604 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:01.064614 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:01.064683 (+ 69us) server_negotiation.cc:407] Connection header received
0924 04:59:01.065967 (+ 1284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:01.065979 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:01.066030 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:01.066154 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:01.066818 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:01.068032 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:01.069122 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:01.069658 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:01.069761 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:01.072162 (+ 2401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:01.072166 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:01.072167 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:59:01.072177 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:01.073699 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:01.074004 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:01.074007 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:01.074008 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:01.074111 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:01.074295 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:01.074295 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:01.074296 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:01.074522 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:01.074666 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:01.074759 (+ 93us) server_negotiation.cc:299] Negotiation successful
0924 04:59:01.075001 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":79,"threads_started":1}
I20260924 04:59:01.090344 2181 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:59:01.091074 2181 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:59:01.095518 2181 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cb927924e0fe40f185c2f20be51a64f0: Generated new TSK 0
I20260924 04:59:01.095690 2181 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225929848825-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225929848825-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:59:01.120111 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225929848825-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225929848825-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225929848825-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225929848825-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42637
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225929848825-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:59:01.198894 2193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:59:01.199097 2193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:59:01.199120 2193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:59:01.200242 2193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:59:01.200289 2193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:59:01.200313 2193 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 04:59:01.200336 2193 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 04:59:01.200366 2193 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:59:01.201814 2193 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225929848825-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225929848825-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225929848825-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225929848825-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:42637
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2193
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225929848825-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 04:59:01.202031 2193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:59:01.202237 2193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:59:01.206864 2201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:59:01.206887 2198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:59:01.206885 2199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:59:01.207046 2193 server_base.cc:1034] running on GCE node
I20260924 04:59:01.207265 2193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:59:01.207482 2193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:59:01.208626 2193 hybrid_clock.cc:648] HybridClock initialized: now 1790225941208612 us; error 30 us; skew 500 ppm
Sep 24 04:59:01 dist-test-slave-70st krb5kdc[2076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225941, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:59:01.210960 2193 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:59:01.211357 2193 webserver.cc:460] Webserver started at http://127.0.98.1:39779/ using document root <none> and password file <none>
I20260924 04:59:01.211571 2193 fs_manager.cc:359] Metadata directory not provided
I20260924 04:59:01.211619 2193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:59:01.211740 2193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:59:01.212724 2193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225929848825-392-0/minicluster-data/ts-0/data/instance:
uuid: "2a10411e514d42a6b1e9fa6d6347eb83"
format_stamp: "Formatted at 2026-09-24 04:59:01 on dist-test-slave-70st"
server_key: "3b12f2544aa9f25da9cf31f01ed540b1"
server_key_iv: "b1d16fb736ef9e15a541b26e760d6db0"
server_key_version: "encryptionkey@0"
I20260924 04:59:01.213048 2193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225929848825-392-0/minicluster-data/ts-0/wal/instance:
uuid: "2a10411e514d42a6b1e9fa6d6347eb83"
format_stamp: "Formatted at 2026-09-24 04:59:01 on dist-test-slave-70st"
server_key: "3b12f2544aa9f25da9cf31f01ed540b1"
server_key_iv: "b1d16fb736ef9e15a541b26e760d6db0"
server_key_version: "encryptionkey@0"
I20260924 04:59:01.214409 2193 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 04:59:01.215194 2208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:01.215375 2193 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 04:59:01.215459 2193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225929848825-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225929848825-392-0/minicluster-data/ts-0/wal
uuid: "2a10411e514d42a6b1e9fa6d6347eb83"
format_stamp: "Formatted at 2026-09-24 04:59:01 on dist-test-slave-70st"
server_key: "3b12f2544aa9f25da9cf31f01ed540b1"
server_key_iv: "b1d16fb736ef9e15a541b26e760d6db0"
server_key_version: "encryptionkey@0"
I20260924 04:59:01.215521 2193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225929848825-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225929848825-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225929848825-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:59:01.237258 2193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:59:01.239264 2193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:59:01.239410 2193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:59:01.239903 2193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:59:01.239946 2193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:01.239980 2193 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:59:01.239997 2193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:01.246315 2193 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38697
I20260924 04:59:01.246446 2323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38697 every 8 connection(s)
I20260924 04:59:01.246691 2193 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225929848825-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 04:59:01 dist-test-slave-70st krb5kdc[2076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225941, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:59:01.249543 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:01.240411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55919 (local address 127.0.98.62:42637)
0924 04:59:01.240459 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:01.240461 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:01.240704 (+ 243us) server_negotiation.cc:407] Connection header received
0924 04:59:01.241241 (+ 537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:01.241243 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:01.241257 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:01.241326 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:01.242252 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:01.243023 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:01.244511 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:01.244785 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:01.244845 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:01.247358 (+ 2513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:01.247360 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:01.247360 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:59:01.247366 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:01.248438 (+ 1072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:01.248800 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:01.248801 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:01.248801 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:01.248843 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:01.249023 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:01.249024 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:01.249024 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:01.249113 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:01.249156 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:01.249356 (+ 200us) server_negotiation.cc:299] Negotiation successful
0924 04:59:01.249464 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 04:59:01.249691 2217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:01.240409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42637 (local address 127.0.98.1:55919)
0924 04:59:01.240582 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:01.240618 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:01.241112 (+ 494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:01.241522 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:01.241528 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:01.241883 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:01.242161 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:01.242168 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:01.243122 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:01.243123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:01.244431 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:01.244434 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:01.244883 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:01.244885 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:01.245033 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:01.245603 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:01.245614 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:01.247272 (+ 1658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:59:01.248525 (+ 1253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:01.248532 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:01.248533 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:01.248740 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:01.248881 (+ 141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:01.248882 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:01.248882 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:01.248966 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:01.249158 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:59:01.249161 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:59:01.249271 (+ 110us) client_negotiation.cc:769] Sending connection context
0924 04:59:01.249344 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 04:59:01.249498 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":98,"threads_started":1}
I20260924 04:59:01.250150 2324 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42637
I20260924 04:59:01.250236 2324 heartbeater.cc:464] Registering TS with master...
I20260924 04:59:01.250420 2324 heartbeater.cc:511] Master 127.0.98.62:42637 requested a full tablet report, sending...
I20260924 04:59:01.251868 2120 ts_manager.cc:194] Registered new tserver with Master: 2a10411e514d42a6b1e9fa6d6347eb83 (127.0.98.1:38697)
I20260924 04:59:01.252667 2120 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:55919
I20260924 04:59:01.254690 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 2193
I20260924 04:59:01.254804 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225929848825-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:59:01.254997 392 external_mini_cluster.cc:1428] Setting key 1138d87e6083d87783e51bda34ff6a9b
I20260924 04:59:01.256488 2330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:01.251126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42637 (local address 127.0.98.1:41533)
0924 04:59:01.251251 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:01.251261 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:01.251316 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:01.251436 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:01.251441 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:01.251617 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:01.251671 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:01.251673 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:01.252495 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:01.252496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:01.253074 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:01.253076 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:01.253380 (+ 304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:01.253381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:01.253463 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:01.253817 (+ 354us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:01.253822 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:01.254180 (+ 358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:59:01.255678 (+ 1498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:01.255679 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:01.255679 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:01.255842 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:01.256023 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:01.256023 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:01.256024 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:01.256065 (+ 41us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:01.256261 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:59:01.256262 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:59:01.256304 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 04:59:01.256349 (+ 45us) client_negotiation.cc:240] Negotiation successful
0924 04:59:01.256418 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":77,"threads_started":1}
I20260924 04:59:01.256515 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:01.251210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41533 (local address 127.0.98.62:42637)
0924 04:59:01.251251 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:01.251253 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:01.251305 (+ 52us) server_negotiation.cc:407] Connection header received
0924 04:59:01.251365 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:01.251365 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:01.251390 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:01.251431 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:01.251727 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:01.252418 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:01.253142 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:01.253317 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:01.253366 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:01.254260 (+ 894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:01.254261 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:01.254261 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:59:01.254267 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:01.255551 (+ 1284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:01.255910 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:01.255911 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:01.255911 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:01.255964 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:01.256108 (+ 144us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:01.256109 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:01.256109 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:01.256217 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:01.256272 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:01.256370 (+ 98us) server_negotiation.cc:299] Negotiation successful
0924 04:59:01.256437 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225929848825-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225929848825-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:59:01.278431 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225929848825-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225929848825-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225929848825-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225929848825-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42637
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225929848825-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:59:01.355834 2333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:59:01.356030 2333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:59:01.356053 2333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:59:01.357157 2333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:59:01.357196 2333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:59:01.357215 2333 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 04:59:01.357239 2333 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 04:59:01.357268 2333 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:59:01.358702 2333 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225929848825-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225929848825-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225929848825-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225929848825-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:42637
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2333
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225929848825-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 04:59:01.358916 2333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:59:01.359115 2333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:59:01.363667 2333 server_base.cc:1034] running on GCE node
W20260924 04:59:01.363682 2341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:59:01.363756 2339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:59:01.363865 2338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:59:01.364110 2333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:59:01.364333 2333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:59:01.365463 2333 hybrid_clock.cc:648] HybridClock initialized: now 1790225941365461 us; error 45 us; skew 500 ppm
Sep 24 04:59:01 dist-test-slave-70st krb5kdc[2076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225941, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:59:01.367755 2333 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:59:01.368191 2333 webserver.cc:460] Webserver started at http://127.0.98.2:38163/ using document root <none> and password file <none>
I20260924 04:59:01.368468 2333 fs_manager.cc:359] Metadata directory not provided
I20260924 04:59:01.368515 2333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:59:01.368605 2333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:59:01.369628 2333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225929848825-392-0/minicluster-data/ts-1/data/instance:
uuid: "2e670717494940d193cc73c7ec81a78b"
format_stamp: "Formatted at 2026-09-24 04:59:01 on dist-test-slave-70st"
server_key: "537a56d7c16711480fc8d33b56c21365"
server_key_iv: "31b2d0522732314053c07f25a2cd460b"
server_key_version: "encryptionkey@0"
I20260924 04:59:01.369913 2333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225929848825-392-0/minicluster-data/ts-1/wal/instance:
uuid: "2e670717494940d193cc73c7ec81a78b"
format_stamp: "Formatted at 2026-09-24 04:59:01 on dist-test-slave-70st"
server_key: "537a56d7c16711480fc8d33b56c21365"
server_key_iv: "31b2d0522732314053c07f25a2cd460b"
server_key_version: "encryptionkey@0"
I20260924 04:59:01.371050 2333 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 04:59:01.371644 2348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:01.371794 2333 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 04:59:01.371873 2333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225929848825-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225929848825-392-0/minicluster-data/ts-1/wal
uuid: "2e670717494940d193cc73c7ec81a78b"
format_stamp: "Formatted at 2026-09-24 04:59:01 on dist-test-slave-70st"
server_key: "537a56d7c16711480fc8d33b56c21365"
server_key_iv: "31b2d0522732314053c07f25a2cd460b"
server_key_version: "encryptionkey@0"
I20260924 04:59:01.371930 2333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225929848825-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225929848825-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225929848825-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:59:01.405211 2333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:59:01.407122 2333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:59:01.407258 2333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:59:01.407758 2333 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:59:01.407800 2333 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:01.407830 2333 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:59:01.407860 2333 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 04:59:01 dist-test-slave-70st krb5kdc[2076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225941, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:59:01.414323 2333 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36443
I20260924 04:59:01.414381 2463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36443 every 8 connection(s)
I20260924 04:59:01.414714 2333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225929848825-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:59:01.417347 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:01.408151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35937 (local address 127.0.98.62:42637)
0924 04:59:01.408214 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:01.408216 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:01.408385 (+ 169us) server_negotiation.cc:407] Connection header received
0924 04:59:01.408852 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:01.408854 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:01.408866 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:01.408942 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:01.409575 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:01.410067 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:01.411110 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:01.411307 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:01.411362 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:01.413724 (+ 2362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:01.413726 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:01.413727 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:59:01.413739 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:01.415200 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:01.416644 (+ 1444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:01.416645 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:01.416646 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:01.416703 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:01.416892 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:01.416892 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:01.416892 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:01.416995 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:01.417039 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:01.417228 (+ 189us) server_negotiation.cc:299] Negotiation successful
0924 04:59:01.417284 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 04:59:01.417562 2357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:01.408111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42637 (local address 127.0.98.2:35937)
0924 04:59:01.408265 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:01.408305 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:01.408763 (+ 458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:01.409008 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:01.409013 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:01.409267 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:01.409492 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:01.409498 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:01.410147 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:01.410149 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:01.411036 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:01.411039 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:01.411372 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:01.411373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:01.411489 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:01.411945 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:01.411954 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:01.413649 (+ 1695us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:59:01.415253 (+ 1604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:01.415256 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:01.415258 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:01.416557 (+ 1299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:01.416752 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:01.416752 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:01.416752 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:01.416850 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:01.417029 (+ 179us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:59:01.417032 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:59:01.417138 (+ 106us) client_negotiation.cc:769] Sending connection context
0924 04:59:01.417207 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 04:59:01.417329 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":88,"threads_started":1}
I20260924 04:59:01.417971 2464 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42637
I20260924 04:59:01.418049 2464 heartbeater.cc:464] Registering TS with master...
I20260924 04:59:01.418211 2464 heartbeater.cc:511] Master 127.0.98.62:42637 requested a full tablet report, sending...
I20260924 04:59:01.418553 2121 ts_manager.cc:194] Registered new tserver with Master: 2e670717494940d193cc73c7ec81a78b (127.0.98.2:36443)
I20260924 04:59:01.418926 2121 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35937
I20260924 04:59:01.422900 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 2333
I20260924 04:59:01.422976 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225929848825-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:59:01.423170 392 external_mini_cluster.cc:1428] Setting key 79507cfdeb4d3b6225e2f9117ce8394f
I20260924 04:59:01.423645 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:01.419011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49853 (local address 127.0.98.62:42637)
0924 04:59:01.419047 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:01.419049 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:01.419129 (+ 80us) server_negotiation.cc:407] Connection header received
0924 04:59:01.419213 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:01.419214 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:01.419224 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:01.419277 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:01.419597 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:01.420052 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:01.420667 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:01.420842 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:01.420889 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:01.421609 (+ 720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:01.421610 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:01.421610 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:59:01.421616 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:01.422650 (+ 1034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:01.422937 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:01.422938 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:01.422938 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:01.422992 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:01.423181 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:01.423181 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:01.423182 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:01.423345 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:01.423406 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:01.423529 (+ 123us) server_negotiation.cc:299] Negotiation successful
0924 04:59:01.423588 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 04:59:01.423694 2470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:01.418937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42637 (local address 127.0.98.2:49853)
0924 04:59:01.419053 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:01.419063 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:01.419118 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:01.419283 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:01.419283 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:01.419461 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:01.419530 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:01.419531 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:01.420123 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:01.420123 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:01.420603 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:01.420605 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:01.420893 (+ 288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:01.420893 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:01.420941 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:01.421265 (+ 324us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:01.421268 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:01.421553 (+ 285us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:59:01.422721 (+ 1168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:01.422722 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:01.422722 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:01.422866 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:01.423049 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:01.423050 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:01.423050 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:01.423119 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:01.423401 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:59:01.423402 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:59:01.423444 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 04:59:01.423507 (+ 63us) client_negotiation.cc:240] Negotiation successful
0924 04:59:01.423598 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":64,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225929848825-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225929848825-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:59:01.446055 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225929848825-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225929848825-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225929848825-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225929848825-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42637
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225929848825-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:59:01.523151 2473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:59:01.523346 2473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:59:01.523379 2473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:59:01.524487 2473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:59:01.524528 2473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:59:01.524554 2473 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 04:59:01.524569 2473 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 04:59:01.524592 2473 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:59:01.526023 2473 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225929848825-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225929848825-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225929848825-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225929848825-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:42637
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2473
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225929848825-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 04:59:01.526248 2473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:59:01.526459 2473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:59:01.531008 2478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:59:01.531124 2473 server_base.cc:1034] running on GCE node
W20260924 04:59:01.531010 2481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:59:01.531068 2479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:59:01.531430 2473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:59:01.531642 2473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:59:01.532785 2473 hybrid_clock.cc:648] HybridClock initialized: now 1790225941532776 us; error 31 us; skew 500 ppm
Sep 24 04:59:01 dist-test-slave-70st krb5kdc[2076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225941, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:59:01.535097 2473 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:59:01.535519 2473 webserver.cc:460] Webserver started at http://127.0.98.3:36619/ using document root <none> and password file <none>
I20260924 04:59:01.535717 2473 fs_manager.cc:359] Metadata directory not provided
I20260924 04:59:01.535763 2473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:59:01.535880 2473 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:59:01.536880 2473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225929848825-392-0/minicluster-data/ts-2/data/instance:
uuid: "ff5ba82680c5444aa7f89f73e77c4504"
format_stamp: "Formatted at 2026-09-24 04:59:01 on dist-test-slave-70st"
server_key: "842057bdf188410a146100b6781986d1"
server_key_iv: "0610b372a1670c781dcd249b689b4acf"
server_key_version: "encryptionkey@0"
I20260924 04:59:01.537192 2473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225929848825-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ff5ba82680c5444aa7f89f73e77c4504"
format_stamp: "Formatted at 2026-09-24 04:59:01 on dist-test-slave-70st"
server_key: "842057bdf188410a146100b6781986d1"
server_key_iv: "0610b372a1670c781dcd249b689b4acf"
server_key_version: "encryptionkey@0"
I20260924 04:59:01.538440 2473 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 04:59:01.539063 2488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:01.539201 2473 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 04:59:01.539276 2473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225929848825-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225929848825-392-0/minicluster-data/ts-2/wal
uuid: "ff5ba82680c5444aa7f89f73e77c4504"
format_stamp: "Formatted at 2026-09-24 04:59:01 on dist-test-slave-70st"
server_key: "842057bdf188410a146100b6781986d1"
server_key_iv: "0610b372a1670c781dcd249b689b4acf"
server_key_version: "encryptionkey@0"
I20260924 04:59:01.539340 2473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225929848825-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225929848825-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225929848825-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:59:01.550787 2473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:59:01.552376 2473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:59:01.552506 2473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:59:01.552953 2473 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:59:01.552991 2473 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:01.553020 2473 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:59:01.553042 2473 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 04:59:01 dist-test-slave-70st krb5kdc[2076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225941, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:59:01.559424 2473 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36967
I20260924 04:59:01.559829 2473 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225929848825-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:59:01.560173 2603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36967 every 8 connection(s)
I20260924 04:59:01.561705 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:01.553337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54893 (local address 127.0.98.62:42637)
0924 04:59:01.553384 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:01.553386 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:01.553600 (+ 214us) server_negotiation.cc:407] Connection header received
0924 04:59:01.554082 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:01.554083 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:01.554096 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:01.554147 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:01.554737 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:01.555239 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:01.556261 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:01.556492 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:01.556548 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:01.558960 (+ 2412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:01.558961 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:01.558961 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:59:01.558969 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:01.560400 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:01.560821 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:01.560822 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:01.560823 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:01.560876 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:01.561092 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:01.561093 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:01.561093 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:01.561197 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:01.561252 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:01.561539 (+ 287us) server_negotiation.cc:299] Negotiation successful
0924 04:59:01.561616 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 04:59:01.561847 2497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:01.553322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42637 (local address 127.0.98.3:54893)
0924 04:59:01.553491 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:01.553525 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:01.553989 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:01.554212 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:01.554215 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:01.554467 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:01.554653 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:01.554657 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:01.555320 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:01.555321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:01.556183 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:01.556186 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:01.556558 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:01.556559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:01.556689 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:01.557159 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:01.557168 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:01.558889 (+ 1721us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:59:01.560468 (+ 1579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:01.560474 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:01.560476 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:01.560757 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:01.560929 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:01.560930 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:01.560931 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:01.561038 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:01.561249 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:59:01.561254 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:59:01.561382 (+ 128us) client_negotiation.cc:769] Sending connection context
0924 04:59:01.561507 (+ 125us) client_negotiation.cc:240] Negotiation successful
0924 04:59:01.561654 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":108,"threads_started":1}
I20260924 04:59:01.562235 2604 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42637
I20260924 04:59:01.562307 2604 heartbeater.cc:464] Registering TS with master...
I20260924 04:59:01.562479 2604 heartbeater.cc:511] Master 127.0.98.62:42637 requested a full tablet report, sending...
I20260924 04:59:01.562781 2120 ts_manager.cc:194] Registered new tserver with Master: ff5ba82680c5444aa7f89f73e77c4504 (127.0.98.3:36967)
I20260924 04:59:01.563102 2120 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:54893
I20260924 04:59:01.567687 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:01.563168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49531 (local address 127.0.98.62:42637)
0924 04:59:01.563260 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:01.563261 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:01.563269 (+ 8us) server_negotiation.cc:407] Connection header received
0924 04:59:01.563325 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:01.563325 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:01.563332 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:01.563376 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:01.563662 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:01.564111 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:01.564770 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:01.564947 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:01.564996 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:01.565768 (+ 772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:01.565770 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:01.565770 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:59:01.565776 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:01.566745 (+ 969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:01.567023 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:01.567024 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:01.567024 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:01.567069 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:01.567222 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:01.567223 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:01.567223 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:01.567317 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:01.567369 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:01.567508 (+ 139us) server_negotiation.cc:299] Negotiation successful
0924 04:59:01.567598 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 04:59:01.567700 2610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:01.563105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42637 (local address 127.0.98.3:49531)
0924 04:59:01.563217 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:01.563226 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:01.563271 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:01.563378 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:01.563379 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:01.563524 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:01.563579 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:01.563580 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:01.564193 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:01.564194 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:01.564702 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:01.564704 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:01.565006 (+ 302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:01.565007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:01.565077 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:01.565398 (+ 321us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:01.565401 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:01.565697 (+ 296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:59:01.566804 (+ 1107us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:01.566805 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:01.566806 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:01.566957 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:01.567121 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:01.567122 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:01.567122 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:01.567175 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:01.567369 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:59:01.567370 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:59:01.567414 (+ 44us) client_negotiation.cc:769] Sending connection context
0924 04:59:01.567474 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 04:59:01.567600 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":68,"threads_started":1}
I20260924 04:59:01.570001 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 2473
I20260924 04:59:01.570096 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225929848825-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:59:01.570281 392 external_mini_cluster.cc:1428] Setting key ae0a7d97dba26b203e4b2a9c5233acfb
I20260924 04:59:01.571007 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 04:59:01 dist-test-slave-70st krb5kdc[2076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225941, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 04:59:01 dist-test-slave-70st krb5kdc[2076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225941, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:59:01.591706 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:01.585472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59626 (local address 127.0.98.62:42637)
0924 04:59:01.585516 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:01.585518 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:01.585527 (+ 9us) server_negotiation.cc:407] Connection header received
0924 04:59:01.585599 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:01.585599 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:01.585609 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:01.585647 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:01.586173 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:01.586670 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:01.587372 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:01.587565 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:01.587627 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:01.589778 (+ 2151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:01.589779 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:01.589780 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:59:01.589788 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:01.590893 (+ 1105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:01.591177 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:01.591177 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:01.591178 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:01.591220 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:01.591346 (+ 126us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:01.591346 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:01.591346 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:01.591436 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:01.591505 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:01.591584 (+ 79us) server_negotiation.cc:299] Negotiation successful
0924 04:59:01.591639 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 04:59:01.593204 2120 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59626:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:59:01.594782 2120 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:59:01.600585 2619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:01.596828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36443 (local address 127.0.0.1:43346)
0924 04:59:01.597124 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:01.597131 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:01.597163 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:01.597328 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:01.597331 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:01.597333 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:59:01.597398 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:01.597400 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:01.598377 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:01.598378 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:01.599324 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:01.599326 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:01.600294 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:01.600295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:01.600383 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:01.600397 (+ 14us) client_negotiation.cc:769] Sending connection context
0924 04:59:01.600464 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 04:59:01.600509 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"mutex_wait_us":2,"thread_start_us":53,"threads_started":1}
I20260924 04:59:01.600646 2621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:01.597026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43346 (local address 127.0.98.2:36443)
0924 04:59:01.597170 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:01.597175 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:01.597185 (+ 10us) server_negotiation.cc:407] Connection header received
0924 04:59:01.597222 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:01.597224 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:01.597236 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:01.597311 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:59:01.597495 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:01.598302 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:01.599420 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:01.600119 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:01.600201 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:01.600260 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:01.600502 (+ 242us) server_negotiation.cc:299] Negotiation successful
0924 04:59:01.600541 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":71,"threads_started":1}
I20260924 04:59:01.600924 2618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:01.596827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38697 (local address 127.0.0.1:50284)
0924 04:59:01.596929 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:01.596969 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:01.597034 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:01.597521 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:01.597521 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:01.597524 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:59:01.597632 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:01.597634 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:01.598888 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:01.598889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:01.599839 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:01.599841 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:01.600660 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:01.600660 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:01.600767 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:01.600771 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 04:59:01.600822 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 04:59:01.600859 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78,"thread_start_us":54,"threads_started":1}
I20260924 04:59:01.600973 2622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:01.597211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50284 (local address 127.0.98.1:38697)
0924 04:59:01.597371 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:01.597375 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:01.597384 (+ 9us) server_negotiation.cc:407] Connection header received
0924 04:59:01.597435 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:01.597437 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:01.597451 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:01.597527 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:59:01.597715 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:01.598800 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:01.599921 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:01.600507 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:01.600567 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:01.600625 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:01.600848 (+ 223us) server_negotiation.cc:299] Negotiation successful
0924 04:59:01.600891 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":74,"threads_started":1}
I20260924 04:59:01.601081 2620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:01.596828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36967 (local address 127.0.0.1:44996)
0924 04:59:01.597048 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:01.597056 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:01.597094 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:01.597778 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:01.597778 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:01.597780 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:59:01.597840 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:01.597842 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:01.599020 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:01.599021 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:01.600001 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:01.600003 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:01.600884 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:01.600885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:01.600944 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:01.600948 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 04:59:01.600976 (+ 28us) client_negotiation.cc:240] Negotiation successful
0924 04:59:01.601016 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"spinlock_wait_cycles":896,"thread_start_us":62,"threads_started":1}
I20260924 04:59:01.601438 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:01.597382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44996 (local address 127.0.98.3:36967)
0924 04:59:01.597625 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:01.597629 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:01.597638 (+ 9us) server_negotiation.cc:407] Connection header received
0924 04:59:01.597673 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:01.597675 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:01.597690 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:01.597907 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:59:01.597965 (+ 58us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:01.598943 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:01.600078 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:01.600799 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:01.601182 (+ 383us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:01.601250 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:01.601272 (+ 22us) server_negotiation.cc:299] Negotiation successful
0924 04:59:01.601322 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":140,"threads_started":1}
I20260924 04:59:01.601626 2398 tablet_service.cc:1467] Processing CreateTablet for tablet 9290b43630754f14abfdda9b1d780d27 (DEFAULT_TABLE table=test-table [id=fdfd922e0c7b476787f5d9826af5a3c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:59:01.601681 2258 tablet_service.cc:1467] Processing CreateTablet for tablet 9290b43630754f14abfdda9b1d780d27 (DEFAULT_TABLE table=test-table [id=fdfd922e0c7b476787f5d9826af5a3c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:59:01.601995 2258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9290b43630754f14abfdda9b1d780d27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:59:01.602000 2398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9290b43630754f14abfdda9b1d780d27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:59:01.602003 2538 tablet_service.cc:1467] Processing CreateTablet for tablet 9290b43630754f14abfdda9b1d780d27 (DEFAULT_TABLE table=test-table [id=fdfd922e0c7b476787f5d9826af5a3c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:59:01.602311 2538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9290b43630754f14abfdda9b1d780d27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:59:01.604727 2624 tablet_bootstrap.cc:492] T 9290b43630754f14abfdda9b1d780d27 P ff5ba82680c5444aa7f89f73e77c4504: Bootstrap starting.
I20260924 04:59:01.605468 2625 tablet_bootstrap.cc:492] T 9290b43630754f14abfdda9b1d780d27 P 2e670717494940d193cc73c7ec81a78b: Bootstrap starting.
I20260924 04:59:01.605652 2624 tablet_bootstrap.cc:654] T 9290b43630754f14abfdda9b1d780d27 P ff5ba82680c5444aa7f89f73e77c4504: Neither blocks nor log segments found. Creating new log.
I20260924 04:59:01.606078 2625 tablet_bootstrap.cc:654] T 9290b43630754f14abfdda9b1d780d27 P 2e670717494940d193cc73c7ec81a78b: Neither blocks nor log segments found. Creating new log.
I20260924 04:59:01.606130 2624 log.cc:824] T 9290b43630754f14abfdda9b1d780d27 P ff5ba82680c5444aa7f89f73e77c4504: Log is configured to *not* fsync() on all Append() calls
I20260924 04:59:01.606422 2625 log.cc:824] T 9290b43630754f14abfdda9b1d780d27 P 2e670717494940d193cc73c7ec81a78b: Log is configured to *not* fsync() on all Append() calls
I20260924 04:59:01.606694 2626 tablet_bootstrap.cc:492] T 9290b43630754f14abfdda9b1d780d27 P 2a10411e514d42a6b1e9fa6d6347eb83: Bootstrap starting.
I20260924 04:59:01.606895 2624 tablet_bootstrap.cc:492] T 9290b43630754f14abfdda9b1d780d27 P ff5ba82680c5444aa7f89f73e77c4504: No bootstrap required, opened a new log
I20260924 04:59:01.606968 2624 ts_tablet_manager.cc:1397] T 9290b43630754f14abfdda9b1d780d27 P ff5ba82680c5444aa7f89f73e77c4504: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 04:59:01.607012 2625 tablet_bootstrap.cc:492] T 9290b43630754f14abfdda9b1d780d27 P 2e670717494940d193cc73c7ec81a78b: No bootstrap required, opened a new log
I20260924 04:59:01.607079 2625 ts_tablet_manager.cc:1397] T 9290b43630754f14abfdda9b1d780d27 P 2e670717494940d193cc73c7ec81a78b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 04:59:01.607301 2626 tablet_bootstrap.cc:654] T 9290b43630754f14abfdda9b1d780d27 P 2a10411e514d42a6b1e9fa6d6347eb83: Neither blocks nor log segments found. Creating new log.
I20260924 04:59:01.607597 2626 log.cc:824] T 9290b43630754f14abfdda9b1d780d27 P 2a10411e514d42a6b1e9fa6d6347eb83: Log is configured to *not* fsync() on all Append() calls
I20260924 04:59:01.608354 2626 tablet_bootstrap.cc:492] T 9290b43630754f14abfdda9b1d780d27 P 2a10411e514d42a6b1e9fa6d6347eb83: No bootstrap required, opened a new log
I20260924 04:59:01.608417 2626 ts_tablet_manager.cc:1397] T 9290b43630754f14abfdda9b1d780d27 P 2a10411e514d42a6b1e9fa6d6347eb83: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 04:59:01.608734 2625 raft_consensus.cc:348] T 9290b43630754f14abfdda9b1d780d27 P 2e670717494940d193cc73c7ec81a78b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a10411e514d42a6b1e9fa6d6347eb83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } } peers { permanent_uuid: "2e670717494940d193cc73c7ec81a78b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36443 } } peers { permanent_uuid: "ff5ba82680c5444aa7f89f73e77c4504" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36967 } }
I20260924 04:59:01.608865 2625 raft_consensus.cc:374] T 9290b43630754f14abfdda9b1d780d27 P 2e670717494940d193cc73c7ec81a78b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:59:01.608896 2625 raft_consensus.cc:729] T 9290b43630754f14abfdda9b1d780d27 P 2e670717494940d193cc73c7ec81a78b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e670717494940d193cc73c7ec81a78b, State: Initialized, Role: FOLLOWER
I20260924 04:59:01.608803 2624 raft_consensus.cc:348] T 9290b43630754f14abfdda9b1d780d27 P ff5ba82680c5444aa7f89f73e77c4504 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a10411e514d42a6b1e9fa6d6347eb83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } } peers { permanent_uuid: "2e670717494940d193cc73c7ec81a78b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36443 } } peers { permanent_uuid: "ff5ba82680c5444aa7f89f73e77c4504" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36967 } }
I20260924 04:59:01.608950 2624 raft_consensus.cc:374] T 9290b43630754f14abfdda9b1d780d27 P ff5ba82680c5444aa7f89f73e77c4504 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:59:01.608978 2624 raft_consensus.cc:729] T 9290b43630754f14abfdda9b1d780d27 P ff5ba82680c5444aa7f89f73e77c4504 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff5ba82680c5444aa7f89f73e77c4504, State: Initialized, Role: FOLLOWER
I20260924 04:59:01.609028 2625 consensus_queue.cc:260] T 9290b43630754f14abfdda9b1d780d27 P 2e670717494940d193cc73c7ec81a78b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a10411e514d42a6b1e9fa6d6347eb83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } } peers { permanent_uuid: "2e670717494940d193cc73c7ec81a78b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36443 } } peers { permanent_uuid: "ff5ba82680c5444aa7f89f73e77c4504" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36967 } }
I20260924 04:59:01.609107 2624 consensus_queue.cc:260] T 9290b43630754f14abfdda9b1d780d27 P ff5ba82680c5444aa7f89f73e77c4504 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a10411e514d42a6b1e9fa6d6347eb83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } } peers { permanent_uuid: "2e670717494940d193cc73c7ec81a78b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36443 } } peers { permanent_uuid: "ff5ba82680c5444aa7f89f73e77c4504" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36967 } }
I20260924 04:59:01.609274 2625 ts_tablet_manager.cc:1428] T 9290b43630754f14abfdda9b1d780d27 P 2e670717494940d193cc73c7ec81a78b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 04:59:01.609359 2624 ts_tablet_manager.cc:1428] T 9290b43630754f14abfdda9b1d780d27 P ff5ba82680c5444aa7f89f73e77c4504: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 04:59:01.609393 2464 heartbeater.cc:503] Master 127.0.98.62:42637 was elected leader, sending a full tablet report...
I20260924 04:59:01.609457 2604 heartbeater.cc:503] Master 127.0.98.62:42637 was elected leader, sending a full tablet report...
I20260924 04:59:01.610517 2626 raft_consensus.cc:348] T 9290b43630754f14abfdda9b1d780d27 P 2a10411e514d42a6b1e9fa6d6347eb83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a10411e514d42a6b1e9fa6d6347eb83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } } peers { permanent_uuid: "2e670717494940d193cc73c7ec81a78b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36443 } } peers { permanent_uuid: "ff5ba82680c5444aa7f89f73e77c4504" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36967 } }
I20260924 04:59:01.610653 2626 raft_consensus.cc:374] T 9290b43630754f14abfdda9b1d780d27 P 2a10411e514d42a6b1e9fa6d6347eb83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:59:01.610689 2626 raft_consensus.cc:729] T 9290b43630754f14abfdda9b1d780d27 P 2a10411e514d42a6b1e9fa6d6347eb83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a10411e514d42a6b1e9fa6d6347eb83, State: Initialized, Role: FOLLOWER
I20260924 04:59:01.610790 2626 consensus_queue.cc:260] T 9290b43630754f14abfdda9b1d780d27 P 2a10411e514d42a6b1e9fa6d6347eb83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a10411e514d42a6b1e9fa6d6347eb83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } } peers { permanent_uuid: "2e670717494940d193cc73c7ec81a78b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36443 } } peers { permanent_uuid: "ff5ba82680c5444aa7f89f73e77c4504" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36967 } }
I20260924 04:59:01.611011 2626 ts_tablet_manager.cc:1428] T 9290b43630754f14abfdda9b1d780d27 P 2a10411e514d42a6b1e9fa6d6347eb83: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 04:59:01.611039 2324 heartbeater.cc:503] Master 127.0.98.62:42637 was elected leader, sending a full tablet report...
W20260924 04:59:01.665246 2465 tablet.cc:2369] T 9290b43630754f14abfdda9b1d780d27 P 2e670717494940d193cc73c7ec81a78b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:59:01.706507 2630 raft_consensus.cc:482] T 9290b43630754f14abfdda9b1d780d27 P 2e670717494940d193cc73c7ec81a78b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:59:01.706662 2630 raft_consensus.cc:504] T 9290b43630754f14abfdda9b1d780d27 P 2e670717494940d193cc73c7ec81a78b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a10411e514d42a6b1e9fa6d6347eb83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } } peers { permanent_uuid: "2e670717494940d193cc73c7ec81a78b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36443 } } peers { permanent_uuid: "ff5ba82680c5444aa7f89f73e77c4504" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36967 } }
I20260924 04:59:01.706982 2630 leader_election.cc:288] T 9290b43630754f14abfdda9b1d780d27 P 2e670717494940d193cc73c7ec81a78b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a10411e514d42a6b1e9fa6d6347eb83 (127.0.98.1:38697), ff5ba82680c5444aa7f89f73e77c4504 (127.0.98.3:36967)
I20260924 04:59:01.709815 2622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:01.707245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42555 (local address 127.0.98.1:38697)
0924 04:59:01.707284 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:01.707286 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:01.707296 (+ 10us) server_negotiation.cc:407] Connection header received
0924 04:59:01.707362 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:01.707363 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:01.707376 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:01.707430 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:59:01.707616 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:01.708115 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:01.709103 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:01.709486 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:01.709539 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:01.709547 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:01.709702 (+ 155us) server_negotiation.cc:299] Negotiation successful
0924 04:59:01.709738 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 04:59:01.709833 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:01.707342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53777 (local address 127.0.98.3:36967)
0924 04:59:01.707387 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:01.707389 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:01.707401 (+ 12us) server_negotiation.cc:407] Connection header received
0924 04:59:01.707463 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:01.707464 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:01.707475 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:01.707548 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:59:01.707697 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:01.708159 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:01.709095 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:01.709507 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:01.709594 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:01.709605 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:01.709732 (+ 127us) server_negotiation.cc:299] Negotiation successful
0924 04:59:01.709768 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 04:59:01.709833 2357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:01.707163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38697 (local address 127.0.98.2:42555)
0924 04:59:01.707226 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:01.707238 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:01.707286 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:01.707456 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:01.707457 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:01.707459 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:59:01.707538 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:01.707540 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:01.708201 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:01.708202 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:01.709004 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:01.709006 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:01.709557 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:01.709558 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:01.709629 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:01.709633 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 04:59:01.709683 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 04:59:01.709725 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":51}
I20260924 04:59:01.709839 2633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:01.707196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36967 (local address 127.0.98.2:53777)
0924 04:59:01.707363 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:01.707372 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:01.707409 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:01.707549 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:01.707549 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:01.707559 (+ 10us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:59:01.707630 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:01.707632 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:01.708237 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:01.708238 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:01.709003 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:01.709005 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:01.709604 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:01.709605 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:01.709660 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:01.709663 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 04:59:01.709715 (+ 52us) client_negotiation.cc:240] Negotiation successful
0924 04:59:01.709751 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":118,"threads_started":1}
I20260924 04:59:01.710215 2278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9290b43630754f14abfdda9b1d780d27" candidate_uuid: "2e670717494940d193cc73c7ec81a78b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a10411e514d42a6b1e9fa6d6347eb83" is_pre_election: true
I20260924 04:59:01.710317 2558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9290b43630754f14abfdda9b1d780d27" candidate_uuid: "2e670717494940d193cc73c7ec81a78b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5ba82680c5444aa7f89f73e77c4504" is_pre_election: true
I20260924 04:59:01.710378 2278 raft_consensus.cc:2454] T 9290b43630754f14abfdda9b1d780d27 P 2a10411e514d42a6b1e9fa6d6347eb83 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e670717494940d193cc73c7ec81a78b in term 0.
I20260924 04:59:01.710434 2558 raft_consensus.cc:2454] T 9290b43630754f14abfdda9b1d780d27 P ff5ba82680c5444aa7f89f73e77c4504 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e670717494940d193cc73c7ec81a78b in term 0.
I20260924 04:59:01.710594 2349 leader_election.cc:302] T 9290b43630754f14abfdda9b1d780d27 P 2e670717494940d193cc73c7ec81a78b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a10411e514d42a6b1e9fa6d6347eb83, 2e670717494940d193cc73c7ec81a78b; no voters:
I20260924 04:59:01.710708 2630 raft_consensus.cc:2781] T 9290b43630754f14abfdda9b1d780d27 P 2e670717494940d193cc73c7ec81a78b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:59:01.710776 2630 raft_consensus.cc:482] T 9290b43630754f14abfdda9b1d780d27 P 2e670717494940d193cc73c7ec81a78b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:59:01.710803 2630 raft_consensus.cc:3037] T 9290b43630754f14abfdda9b1d780d27 P 2e670717494940d193cc73c7ec81a78b [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:59:01.711506 2630 raft_consensus.cc:504] T 9290b43630754f14abfdda9b1d780d27 P 2e670717494940d193cc73c7ec81a78b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a10411e514d42a6b1e9fa6d6347eb83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } } peers { permanent_uuid: "2e670717494940d193cc73c7ec81a78b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36443 } } peers { permanent_uuid: "ff5ba82680c5444aa7f89f73e77c4504" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36967 } }
I20260924 04:59:01.711639 2630 leader_election.cc:288] T 9290b43630754f14abfdda9b1d780d27 P 2e670717494940d193cc73c7ec81a78b [CANDIDATE]: Term 1 election: Requested vote from peers 2a10411e514d42a6b1e9fa6d6347eb83 (127.0.98.1:38697), ff5ba82680c5444aa7f89f73e77c4504 (127.0.98.3:36967)
I20260924 04:59:01.711777 2278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9290b43630754f14abfdda9b1d780d27" candidate_uuid: "2e670717494940d193cc73c7ec81a78b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a10411e514d42a6b1e9fa6d6347eb83"
I20260924 04:59:01.711800 2558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9290b43630754f14abfdda9b1d780d27" candidate_uuid: "2e670717494940d193cc73c7ec81a78b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5ba82680c5444aa7f89f73e77c4504"
I20260924 04:59:01.711869 2278 raft_consensus.cc:3037] T 9290b43630754f14abfdda9b1d780d27 P 2a10411e514d42a6b1e9fa6d6347eb83 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:59:01.711865 2558 raft_consensus.cc:3037] T 9290b43630754f14abfdda9b1d780d27 P ff5ba82680c5444aa7f89f73e77c4504 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:59:01.712538 2278 raft_consensus.cc:2454] T 9290b43630754f14abfdda9b1d780d27 P 2a10411e514d42a6b1e9fa6d6347eb83 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e670717494940d193cc73c7ec81a78b in term 1.
I20260924 04:59:01.712544 2558 raft_consensus.cc:2454] T 9290b43630754f14abfdda9b1d780d27 P ff5ba82680c5444aa7f89f73e77c4504 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e670717494940d193cc73c7ec81a78b in term 1.
I20260924 04:59:01.712695 2349 leader_election.cc:302] T 9290b43630754f14abfdda9b1d780d27 P 2e670717494940d193cc73c7ec81a78b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a10411e514d42a6b1e9fa6d6347eb83, 2e670717494940d193cc73c7ec81a78b; no voters:
I20260924 04:59:01.712792 2630 raft_consensus.cc:2781] T 9290b43630754f14abfdda9b1d780d27 P 2e670717494940d193cc73c7ec81a78b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:59:01.712951 2630 raft_consensus.cc:686] T 9290b43630754f14abfdda9b1d780d27 P 2e670717494940d193cc73c7ec81a78b [term 1 LEADER]: Becoming Leader. State: Replica: 2e670717494940d193cc73c7ec81a78b, State: Running, Role: LEADER
I20260924 04:59:01.713029 2630 consensus_queue.cc:237] T 9290b43630754f14abfdda9b1d780d27 P 2e670717494940d193cc73c7ec81a78b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a10411e514d42a6b1e9fa6d6347eb83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } } peers { permanent_uuid: "2e670717494940d193cc73c7ec81a78b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36443 } } peers { permanent_uuid: "ff5ba82680c5444aa7f89f73e77c4504" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36967 } }
I20260924 04:59:01.713896 2120 catalog_manager.cc:5495] T 9290b43630754f14abfdda9b1d780d27 P 2e670717494940d193cc73c7ec81a78b reported cstate change: term changed from 0 to 1, leader changed from <none> to 2e670717494940d193cc73c7ec81a78b (127.0.98.2). New cstate: current_term: 1 leader_uuid: "2e670717494940d193cc73c7ec81a78b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a10411e514d42a6b1e9fa6d6347eb83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e670717494940d193cc73c7ec81a78b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36443 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff5ba82680c5444aa7f89f73e77c4504" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36967 } health_report { overall_health: UNKNOWN } } }
W20260924 04:59:01.747402 2325 tablet.cc:2369] T 9290b43630754f14abfdda9b1d780d27 P 2a10411e514d42a6b1e9fa6d6347eb83: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 04:59:01 dist-test-slave-70st krb5kdc[2076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225941, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:59:01.772368 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:01.766153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59640 (local address 127.0.98.62:42637)
0924 04:59:01.766228 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:01.766230 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:01.766319 (+ 89us) server_negotiation.cc:407] Connection header received
0924 04:59:01.766393 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:01.766394 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:01.766403 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:01.766446 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:01.766879 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:01.767409 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:01.768062 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:01.768256 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:01.768319 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:01.770228 (+ 1909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:01.770229 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:01.770230 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:59:01.770236 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:01.771317 (+ 1081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:01.771639 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:01.771640 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:01.771640 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:01.771690 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:01.771840 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:01.771840 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:01.771841 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:01.771940 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:01.771993 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:01.772132 (+ 139us) server_negotiation.cc:299] Negotiation successful
0924 04:59:01.772273 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":78,"server-negotiator.queue_time_us":41}
I20260924 04:59:01.772368 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:01.766080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42637 (local address 127.0.0.1:59640)
0924 04:59:01.766253 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:01.766263 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:01.766330 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:01.766460 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:01.766460 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:01.766763 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:01.766806 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:01.766808 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:01.767480 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:01.767481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:01.767995 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:01.767996 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:01.768318 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:01.768319 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:01.768365 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:01.768857 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:01.768861 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:01.770153 (+ 1292us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:59:01.771387 (+ 1234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:01.771394 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:01.771396 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:01.771578 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:01.771740 (+ 162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:01.771740 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:01.771741 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:01.771794 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:01.771994 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:59:01.772002 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:59:01.772048 (+ 46us) client_negotiation.cc:769] Sending connection context
0924 04:59:01.772105 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 04:59:01.772231 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":2,"thread_start_us":103,"threads_started":1}
I20260924 04:59:01.774447 2120 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59640:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 04:59:01.774648 2120 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:59:01.776072 2538 tablet_service.cc:1467] Processing CreateTablet for tablet caea6fb072274e918ae1156aaf478482 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c48aab5d4d41485eb1c94c4df24670de]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 04:59:01.776062 2258 tablet_service.cc:1467] Processing CreateTablet for tablet caea6fb072274e918ae1156aaf478482 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c48aab5d4d41485eb1c94c4df24670de]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 04:59:01.776224 2258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet caea6fb072274e918ae1156aaf478482. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:59:01.776224 2538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet caea6fb072274e918ae1156aaf478482. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:59:01.776168 2398 tablet_service.cc:1467] Processing CreateTablet for tablet caea6fb072274e918ae1156aaf478482 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c48aab5d4d41485eb1c94c4df24670de]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 04:59:01.776320 2398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet caea6fb072274e918ae1156aaf478482. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:59:01.777685 2626 tablet_bootstrap.cc:492] T caea6fb072274e918ae1156aaf478482 P 2a10411e514d42a6b1e9fa6d6347eb83: Bootstrap starting.
I20260924 04:59:01.777693 2625 tablet_bootstrap.cc:492] T caea6fb072274e918ae1156aaf478482 P 2e670717494940d193cc73c7ec81a78b: Bootstrap starting.
I20260924 04:59:01.778096 2626 tablet_bootstrap.cc:654] T caea6fb072274e918ae1156aaf478482 P 2a10411e514d42a6b1e9fa6d6347eb83: Neither blocks nor log segments found. Creating new log.
I20260924 04:59:01.778100 2625 tablet_bootstrap.cc:654] T caea6fb072274e918ae1156aaf478482 P 2e670717494940d193cc73c7ec81a78b: Neither blocks nor log segments found. Creating new log.
I20260924 04:59:01.778753 2624 tablet_bootstrap.cc:492] T caea6fb072274e918ae1156aaf478482 P ff5ba82680c5444aa7f89f73e77c4504: Bootstrap starting.
I20260924 04:59:01.778767 2625 tablet_bootstrap.cc:492] T caea6fb072274e918ae1156aaf478482 P 2e670717494940d193cc73c7ec81a78b: No bootstrap required, opened a new log
I20260924 04:59:01.778807 2625 ts_tablet_manager.cc:1397] T caea6fb072274e918ae1156aaf478482 P 2e670717494940d193cc73c7ec81a78b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:59:01.778903 2626 tablet_bootstrap.cc:492] T caea6fb072274e918ae1156aaf478482 P 2a10411e514d42a6b1e9fa6d6347eb83: No bootstrap required, opened a new log
I20260924 04:59:01.778954 2626 ts_tablet_manager.cc:1397] T caea6fb072274e918ae1156aaf478482 P 2a10411e514d42a6b1e9fa6d6347eb83: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:59:01.778977 2625 raft_consensus.cc:348] T caea6fb072274e918ae1156aaf478482 P 2e670717494940d193cc73c7ec81a78b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a10411e514d42a6b1e9fa6d6347eb83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } } peers { permanent_uuid: "2e670717494940d193cc73c7ec81a78b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36443 } } peers { permanent_uuid: "ff5ba82680c5444aa7f89f73e77c4504" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36967 } }
I20260924 04:59:01.779069 2625 raft_consensus.cc:374] T caea6fb072274e918ae1156aaf478482 P 2e670717494940d193cc73c7ec81a78b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:59:01.779111 2625 raft_consensus.cc:729] T caea6fb072274e918ae1156aaf478482 P 2e670717494940d193cc73c7ec81a78b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e670717494940d193cc73c7ec81a78b, State: Initialized, Role: FOLLOWER
I20260924 04:59:01.779104 2626 raft_consensus.cc:348] T caea6fb072274e918ae1156aaf478482 P 2a10411e514d42a6b1e9fa6d6347eb83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a10411e514d42a6b1e9fa6d6347eb83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } } peers { permanent_uuid: "2e670717494940d193cc73c7ec81a78b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36443 } } peers { permanent_uuid: "ff5ba82680c5444aa7f89f73e77c4504" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36967 } }
I20260924 04:59:01.779170 2626 raft_consensus.cc:374] T caea6fb072274e918ae1156aaf478482 P 2a10411e514d42a6b1e9fa6d6347eb83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:59:01.779155 2625 consensus_queue.cc:260] T caea6fb072274e918ae1156aaf478482 P 2e670717494940d193cc73c7ec81a78b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a10411e514d42a6b1e9fa6d6347eb83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } } peers { permanent_uuid: "2e670717494940d193cc73c7ec81a78b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36443 } } peers { permanent_uuid: "ff5ba82680c5444aa7f89f73e77c4504" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36967 } }
I20260924 04:59:01.779188 2626 raft_consensus.cc:729] T caea6fb072274e918ae1156aaf478482 P 2a10411e514d42a6b1e9fa6d6347eb83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a10411e514d42a6b1e9fa6d6347eb83, State: Initialized, Role: FOLLOWER
I20260924 04:59:01.779237 2625 ts_tablet_manager.cc:1428] T caea6fb072274e918ae1156aaf478482 P 2e670717494940d193cc73c7ec81a78b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:01.779229 2626 consensus_queue.cc:260] T caea6fb072274e918ae1156aaf478482 P 2a10411e514d42a6b1e9fa6d6347eb83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a10411e514d42a6b1e9fa6d6347eb83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } } peers { permanent_uuid: "2e670717494940d193cc73c7ec81a78b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36443 } } peers { permanent_uuid: "ff5ba82680c5444aa7f89f73e77c4504" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36967 } }
I20260924 04:59:01.779269 2624 tablet_bootstrap.cc:654] T caea6fb072274e918ae1156aaf478482 P ff5ba82680c5444aa7f89f73e77c4504: Neither blocks nor log segments found. Creating new log.
I20260924 04:59:01.779316 2626 ts_tablet_manager.cc:1428] T caea6fb072274e918ae1156aaf478482 P 2a10411e514d42a6b1e9fa6d6347eb83: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 04:59:01.779445 2632 tablet_replica.cc:439] T caea6fb072274e918ae1156aaf478482 P 2a10411e514d42a6b1e9fa6d6347eb83: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a10411e514d42a6b1e9fa6d6347eb83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } } peers { permanent_uuid: "2e670717494940d193cc73c7ec81a78b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36443 } } peers { permanent_uuid: "ff5ba82680c5444aa7f89f73e77c4504" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36967 } } }
I20260924 04:59:01.779531 2632 tablet_replica.cc:442] T caea6fb072274e918ae1156aaf478482 P 2a10411e514d42a6b1e9fa6d6347eb83: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:59:01.779629 2630 tablet_replica.cc:439] T caea6fb072274e918ae1156aaf478482 P 2e670717494940d193cc73c7ec81a78b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a10411e514d42a6b1e9fa6d6347eb83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } } peers { permanent_uuid: "2e670717494940d193cc73c7ec81a78b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36443 } } peers { permanent_uuid: "ff5ba82680c5444aa7f89f73e77c4504" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36967 } } }
I20260924 04:59:01.779704 2630 tablet_replica.cc:442] T caea6fb072274e918ae1156aaf478482 P 2e670717494940d193cc73c7ec81a78b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:59:01.779963 2624 tablet_bootstrap.cc:492] T caea6fb072274e918ae1156aaf478482 P ff5ba82680c5444aa7f89f73e77c4504: No bootstrap required, opened a new log
I20260924 04:59:01.780007 2624 ts_tablet_manager.cc:1397] T caea6fb072274e918ae1156aaf478482 P ff5ba82680c5444aa7f89f73e77c4504: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:59:01.780138 2624 raft_consensus.cc:348] T caea6fb072274e918ae1156aaf478482 P ff5ba82680c5444aa7f89f73e77c4504 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a10411e514d42a6b1e9fa6d6347eb83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } } peers { permanent_uuid: "2e670717494940d193cc73c7ec81a78b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36443 } } peers { permanent_uuid: "ff5ba82680c5444aa7f89f73e77c4504" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36967 } }
I20260924 04:59:01.780207 2624 raft_consensus.cc:374] T caea6fb072274e918ae1156aaf478482 P ff5ba82680c5444aa7f89f73e77c4504 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:59:01.780228 2624 raft_consensus.cc:729] T caea6fb072274e918ae1156aaf478482 P ff5ba82680c5444aa7f89f73e77c4504 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff5ba82680c5444aa7f89f73e77c4504, State: Initialized, Role: FOLLOWER
I20260924 04:59:01.780272 2624 consensus_queue.cc:260] T caea6fb072274e918ae1156aaf478482 P ff5ba82680c5444aa7f89f73e77c4504 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a10411e514d42a6b1e9fa6d6347eb83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } } peers { permanent_uuid: "2e670717494940d193cc73c7ec81a78b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36443 } } peers { permanent_uuid: "ff5ba82680c5444aa7f89f73e77c4504" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36967 } }
I20260924 04:59:01.780375 2624 ts_tablet_manager.cc:1428] T caea6fb072274e918ae1156aaf478482 P ff5ba82680c5444aa7f89f73e77c4504: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 04:59:01.780413 2631 tablet_replica.cc:439] T caea6fb072274e918ae1156aaf478482 P ff5ba82680c5444aa7f89f73e77c4504: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a10411e514d42a6b1e9fa6d6347eb83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } } peers { permanent_uuid: "2e670717494940d193cc73c7ec81a78b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36443 } } peers { permanent_uuid: "ff5ba82680c5444aa7f89f73e77c4504" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36967 } } }
I20260924 04:59:01.780479 2631 tablet_replica.cc:442] T caea6fb072274e918ae1156aaf478482 P ff5ba82680c5444aa7f89f73e77c4504: This TxnStatusTablet replica's current role is: FOLLOWER
W20260924 04:59:01.810336 2605 tablet.cc:2369] T 9290b43630754f14abfdda9b1d780d27 P ff5ba82680c5444aa7f89f73e77c4504: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:59:01.899971 2631 raft_consensus.cc:482] T caea6fb072274e918ae1156aaf478482 P ff5ba82680c5444aa7f89f73e77c4504 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:59:01.900068 2631 raft_consensus.cc:504] T caea6fb072274e918ae1156aaf478482 P ff5ba82680c5444aa7f89f73e77c4504 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a10411e514d42a6b1e9fa6d6347eb83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } } peers { permanent_uuid: "2e670717494940d193cc73c7ec81a78b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36443 } } peers { permanent_uuid: "ff5ba82680c5444aa7f89f73e77c4504" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36967 } }
I20260924 04:59:01.900420 2631 leader_election.cc:288] T caea6fb072274e918ae1156aaf478482 P ff5ba82680c5444aa7f89f73e77c4504 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a10411e514d42a6b1e9fa6d6347eb83 (127.0.98.1:38697), 2e670717494940d193cc73c7ec81a78b (127.0.98.2:36443)
I20260924 04:59:01.903084 2622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:01.900712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50109 (local address 127.0.98.1:38697)
0924 04:59:01.900753 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:01.900754 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:01.900762 (+ 8us) server_negotiation.cc:407] Connection header received
0924 04:59:01.900791 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:01.900792 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:01.900799 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:01.900853 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:59:01.901011 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:01.901533 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:01.902432 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:01.902775 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:01.902834 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:01.902844 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:01.902978 (+ 134us) server_negotiation.cc:299] Negotiation successful
0924 04:59:01.903012 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 04:59:01.903091 2497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:01.900603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38697 (local address 127.0.98.3:50109)
0924 04:59:01.900649 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:01.900661 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:01.900706 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:01.900869 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:01.900870 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:01.900872 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:59:01.900943 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:01.900945 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:01.901606 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:01.901606 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:01.902344 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:01.902346 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:01.902840 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:01.902841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:01.902913 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:01.902916 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 04:59:01.902961 (+ 45us) client_negotiation.cc:240] Negotiation successful
0924 04:59:01.903008 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260924 04:59:01.903095 2621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:01.900667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56467 (local address 127.0.98.2:36443)
0924 04:59:01.900712 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:01.900714 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:01.900876 (+ 162us) server_negotiation.cc:407] Connection header received
0924 04:59:01.900907 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:01.900908 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:01.900920 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:01.900967 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:59:01.901102 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:01.901593 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:01.902439 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:01.902787 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:01.902842 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:01.902852 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:01.902996 (+ 144us) server_negotiation.cc:299] Negotiation successful
0924 04:59:01.903032 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 04:59:01.903113 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:01.900603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36443 (local address 127.0.98.3:56467)
0924 04:59:01.900807 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:01.900816 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:01.900860 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:01.900968 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:01.900969 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:01.900971 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:59:01.901042 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:01.901044 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:01.901660 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:01.901661 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:01.902359 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:01.902361 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:01.902871 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:01.902872 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:01.902939 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:01.902941 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 04:59:01.902985 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 04:59:01.903021 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"mutex_wait_us":57,"thread_start_us":105,"threads_started":1}
I20260924 04:59:01.903362 2278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caea6fb072274e918ae1156aaf478482" candidate_uuid: "ff5ba82680c5444aa7f89f73e77c4504" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a10411e514d42a6b1e9fa6d6347eb83" is_pre_election: true
I20260924 04:59:01.903378 2418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caea6fb072274e918ae1156aaf478482" candidate_uuid: "ff5ba82680c5444aa7f89f73e77c4504" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e670717494940d193cc73c7ec81a78b" is_pre_election: true
I20260924 04:59:01.903446 2278 raft_consensus.cc:2454] T caea6fb072274e918ae1156aaf478482 P 2a10411e514d42a6b1e9fa6d6347eb83 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff5ba82680c5444aa7f89f73e77c4504 in term 0.
I20260924 04:59:01.903496 2418 raft_consensus.cc:2454] T caea6fb072274e918ae1156aaf478482 P 2e670717494940d193cc73c7ec81a78b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff5ba82680c5444aa7f89f73e77c4504 in term 0.
I20260924 04:59:01.903630 2489 leader_election.cc:302] T caea6fb072274e918ae1156aaf478482 P ff5ba82680c5444aa7f89f73e77c4504 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a10411e514d42a6b1e9fa6d6347eb83, ff5ba82680c5444aa7f89f73e77c4504; no voters:
I20260924 04:59:01.903769 2631 raft_consensus.cc:2781] T caea6fb072274e918ae1156aaf478482 P ff5ba82680c5444aa7f89f73e77c4504 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:59:01.903807 2631 raft_consensus.cc:482] T caea6fb072274e918ae1156aaf478482 P ff5ba82680c5444aa7f89f73e77c4504 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:59:01.903820 2631 raft_consensus.cc:3037] T caea6fb072274e918ae1156aaf478482 P ff5ba82680c5444aa7f89f73e77c4504 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:59:01.904455 2631 raft_consensus.cc:504] T caea6fb072274e918ae1156aaf478482 P ff5ba82680c5444aa7f89f73e77c4504 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a10411e514d42a6b1e9fa6d6347eb83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } } peers { permanent_uuid: "2e670717494940d193cc73c7ec81a78b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36443 } } peers { permanent_uuid: "ff5ba82680c5444aa7f89f73e77c4504" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36967 } }
I20260924 04:59:01.904577 2631 leader_election.cc:288] T caea6fb072274e918ae1156aaf478482 P ff5ba82680c5444aa7f89f73e77c4504 [CANDIDATE]: Term 1 election: Requested vote from peers 2a10411e514d42a6b1e9fa6d6347eb83 (127.0.98.1:38697), 2e670717494940d193cc73c7ec81a78b (127.0.98.2:36443)
I20260924 04:59:01.904696 2278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caea6fb072274e918ae1156aaf478482" candidate_uuid: "ff5ba82680c5444aa7f89f73e77c4504" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a10411e514d42a6b1e9fa6d6347eb83"
I20260924 04:59:01.904706 2418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caea6fb072274e918ae1156aaf478482" candidate_uuid: "ff5ba82680c5444aa7f89f73e77c4504" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e670717494940d193cc73c7ec81a78b"
I20260924 04:59:01.904767 2418 raft_consensus.cc:3037] T caea6fb072274e918ae1156aaf478482 P 2e670717494940d193cc73c7ec81a78b [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:59:01.904769 2278 raft_consensus.cc:3037] T caea6fb072274e918ae1156aaf478482 P 2a10411e514d42a6b1e9fa6d6347eb83 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:59:01.905331 2278 raft_consensus.cc:2454] T caea6fb072274e918ae1156aaf478482 P 2a10411e514d42a6b1e9fa6d6347eb83 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff5ba82680c5444aa7f89f73e77c4504 in term 1.
I20260924 04:59:01.905339 2418 raft_consensus.cc:2454] T caea6fb072274e918ae1156aaf478482 P 2e670717494940d193cc73c7ec81a78b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff5ba82680c5444aa7f89f73e77c4504 in term 1.
I20260924 04:59:01.905498 2489 leader_election.cc:302] T caea6fb072274e918ae1156aaf478482 P ff5ba82680c5444aa7f89f73e77c4504 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a10411e514d42a6b1e9fa6d6347eb83, ff5ba82680c5444aa7f89f73e77c4504; no voters:
I20260924 04:59:01.905597 2631 raft_consensus.cc:2781] T caea6fb072274e918ae1156aaf478482 P ff5ba82680c5444aa7f89f73e77c4504 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:59:01.905726 2631 raft_consensus.cc:686] T caea6fb072274e918ae1156aaf478482 P ff5ba82680c5444aa7f89f73e77c4504 [term 1 LEADER]: Becoming Leader. State: Replica: ff5ba82680c5444aa7f89f73e77c4504, State: Running, Role: LEADER
I20260924 04:59:01.905807 2631 consensus_queue.cc:237] T caea6fb072274e918ae1156aaf478482 P ff5ba82680c5444aa7f89f73e77c4504 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a10411e514d42a6b1e9fa6d6347eb83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } } peers { permanent_uuid: "2e670717494940d193cc73c7ec81a78b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36443 } } peers { permanent_uuid: "ff5ba82680c5444aa7f89f73e77c4504" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36967 } }
I20260924 04:59:01.906204 2646 tablet_replica.cc:439] T caea6fb072274e918ae1156aaf478482 P ff5ba82680c5444aa7f89f73e77c4504: TxnStatusTablet state changed. Reason: New leader ff5ba82680c5444aa7f89f73e77c4504. Latest consensus state: current_term: 1 leader_uuid: "ff5ba82680c5444aa7f89f73e77c4504" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a10411e514d42a6b1e9fa6d6347eb83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } } peers { permanent_uuid: "2e670717494940d193cc73c7ec81a78b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36443 } } peers { permanent_uuid: "ff5ba82680c5444aa7f89f73e77c4504" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36967 } } }
I20260924 04:59:01.906291 2646 tablet_replica.cc:442] T caea6fb072274e918ae1156aaf478482 P ff5ba82680c5444aa7f89f73e77c4504: This TxnStatusTablet replica's current role is: LEADER
I20260924 04:59:01.906524 2648 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 04:59:01.906636 2648 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 04:59:01.906626 2120 catalog_manager.cc:5495] T caea6fb072274e918ae1156aaf478482 P ff5ba82680c5444aa7f89f73e77c4504 reported cstate change: term changed from 0 to 1, leader changed from <none> to ff5ba82680c5444aa7f89f73e77c4504 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "ff5ba82680c5444aa7f89f73e77c4504" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a10411e514d42a6b1e9fa6d6347eb83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e670717494940d193cc73c7ec81a78b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff5ba82680c5444aa7f89f73e77c4504" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36967 } health_report { overall_health: HEALTHY } } }
I20260924 04:59:01.924682 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:01.922027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45004 (local address 127.0.98.3:36967)
0924 04:59:01.922077 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:01.922080 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:01.922088 (+ 8us) server_negotiation.cc:407] Connection header received
0924 04:59:01.922117 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:01.922118 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:01.922132 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:01.922186 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:59:01.922324 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:01.922787 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:01.923966 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:01.924160 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:01.924211 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:01.924389 (+ 178us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:59:01.924482 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:59:01.924533 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:01.924590 (+ 57us) server_negotiation.cc:299] Negotiation successful
0924 04:59:01.924607 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 04:59:01.924722 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:01.921765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36967 (local address 127.0.0.1:45004)
0924 04:59:01.921847 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:01.921859 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:01.921900 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:01.922206 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:01.922207 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:01.922209 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 04:59:01.922262 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:01.922265 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:01.922848 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:01.922849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:01.923779 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:01.923782 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:01.924225 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:01.924226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:01.924293 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:01.924311 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 04:59:01.924533 (+ 222us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 04:59:01.924539 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 04:59:01.924587 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 04:59:01.924637 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260924 04:59:01.926410 2418 raft_consensus.cc:1261] T caea6fb072274e918ae1156aaf478482 P 2e670717494940d193cc73c7ec81a78b [term 1 FOLLOWER]: Refusing update from remote peer ff5ba82680c5444aa7f89f73e77c4504: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:59:01.926443 2278 raft_consensus.cc:1261] T caea6fb072274e918ae1156aaf478482 P 2a10411e514d42a6b1e9fa6d6347eb83 [term 1 FOLLOWER]: Refusing update from remote peer ff5ba82680c5444aa7f89f73e77c4504: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:59:01.926730 2631 consensus_queue.cc:1035] T caea6fb072274e918ae1156aaf478482 P ff5ba82680c5444aa7f89f73e77c4504 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a10411e514d42a6b1e9fa6d6347eb83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:59:01.926859 2646 consensus_queue.cc:1035] T caea6fb072274e918ae1156aaf478482 P ff5ba82680c5444aa7f89f73e77c4504 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e670717494940d193cc73c7ec81a78b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:59:01.927989 2630 tablet_replica.cc:439] T caea6fb072274e918ae1156aaf478482 P 2e670717494940d193cc73c7ec81a78b: TxnStatusTablet state changed. Reason: New leader ff5ba82680c5444aa7f89f73e77c4504. Latest consensus state: current_term: 1 leader_uuid: "ff5ba82680c5444aa7f89f73e77c4504" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a10411e514d42a6b1e9fa6d6347eb83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } } peers { permanent_uuid: "2e670717494940d193cc73c7ec81a78b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36443 } } peers { permanent_uuid: "ff5ba82680c5444aa7f89f73e77c4504" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36967 } } }
I20260924 04:59:01.928073 2630 tablet_replica.cc:442] T caea6fb072274e918ae1156aaf478482 P 2e670717494940d193cc73c7ec81a78b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:59:01.928033 2632 tablet_replica.cc:439] T caea6fb072274e918ae1156aaf478482 P 2a10411e514d42a6b1e9fa6d6347eb83: TxnStatusTablet state changed. Reason: New leader ff5ba82680c5444aa7f89f73e77c4504. Latest consensus state: current_term: 1 leader_uuid: "ff5ba82680c5444aa7f89f73e77c4504" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a10411e514d42a6b1e9fa6d6347eb83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } } peers { permanent_uuid: "2e670717494940d193cc73c7ec81a78b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36443 } } peers { permanent_uuid: "ff5ba82680c5444aa7f89f73e77c4504" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36967 } } }
I20260924 04:59:01.928112 2632 tablet_replica.cc:442] T caea6fb072274e918ae1156aaf478482 P 2a10411e514d42a6b1e9fa6d6347eb83: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:59:01.929394 2646 tablet_replica.cc:439] T caea6fb072274e918ae1156aaf478482 P ff5ba82680c5444aa7f89f73e77c4504: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ff5ba82680c5444aa7f89f73e77c4504" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a10411e514d42a6b1e9fa6d6347eb83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } } peers { permanent_uuid: "2e670717494940d193cc73c7ec81a78b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36443 } } peers { permanent_uuid: "ff5ba82680c5444aa7f89f73e77c4504" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36967 } } }
I20260924 04:59:01.929502 2646 tablet_replica.cc:442] T caea6fb072274e918ae1156aaf478482 P ff5ba82680c5444aa7f89f73e77c4504: This TxnStatusTablet replica's current role is: LEADER
I20260924 04:59:01.929646 2652 mvcc.cc:205] Tried to move back new op lower bound from 7332765458128216064 to 7332765458046713856. Current Snapshot: MvccSnapshot[applied={T|T < 7332765458128216064}]
I20260924 04:59:01.929935 2632 tablet_replica.cc:439] T caea6fb072274e918ae1156aaf478482 P 2a10411e514d42a6b1e9fa6d6347eb83: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ff5ba82680c5444aa7f89f73e77c4504" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a10411e514d42a6b1e9fa6d6347eb83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } } peers { permanent_uuid: "2e670717494940d193cc73c7ec81a78b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36443 } } peers { permanent_uuid: "ff5ba82680c5444aa7f89f73e77c4504" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36967 } } }
I20260924 04:59:01.930018 2632 tablet_replica.cc:442] T caea6fb072274e918ae1156aaf478482 P 2a10411e514d42a6b1e9fa6d6347eb83: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:59:01.934055 2646 tablet_replica.cc:439] T caea6fb072274e918ae1156aaf478482 P ff5ba82680c5444aa7f89f73e77c4504: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ff5ba82680c5444aa7f89f73e77c4504" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a10411e514d42a6b1e9fa6d6347eb83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } } peers { permanent_uuid: "2e670717494940d193cc73c7ec81a78b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36443 } } peers { permanent_uuid: "ff5ba82680c5444aa7f89f73e77c4504" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36967 } } }
I20260924 04:59:01.934137 2646 tablet_replica.cc:442] T caea6fb072274e918ae1156aaf478482 P ff5ba82680c5444aa7f89f73e77c4504: This TxnStatusTablet replica's current role is: LEADER
I20260924 04:59:01.934382 2653 mvcc.cc:205] Tried to move back new op lower bound from 7332765458128216064 to 7332765458046713856. Current Snapshot: MvccSnapshot[applied={T|T < 7332765458128216064}]
I20260924 04:59:01.934473 2621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:01.931750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43348 (local address 127.0.98.2:36443)
0924 04:59:01.931785 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:01.931787 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:01.931798 (+ 11us) server_negotiation.cc:407] Connection header received
0924 04:59:01.931864 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:01.931865 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:01.931878 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:01.931940 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:59:01.932056 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:01.932548 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:01.933274 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:01.933518 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:01.933577 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:01.933757 (+ 180us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:59:01.933869 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:59:01.933937 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:01.934340 (+ 403us) server_negotiation.cc:299] Negotiation successful
0924 04:59:01.934362 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 04:59:01.934487 2630 tablet_replica.cc:439] T caea6fb072274e918ae1156aaf478482 P 2e670717494940d193cc73c7ec81a78b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ff5ba82680c5444aa7f89f73e77c4504" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a10411e514d42a6b1e9fa6d6347eb83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } } peers { permanent_uuid: "2e670717494940d193cc73c7ec81a78b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36443 } } peers { permanent_uuid: "ff5ba82680c5444aa7f89f73e77c4504" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36967 } } }
I20260924 04:59:01.934547 2630 tablet_replica.cc:442] T caea6fb072274e918ae1156aaf478482 P 2e670717494940d193cc73c7ec81a78b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:59:01.939086 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:01.936985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43037 (local address 127.0.98.3:36967)
0924 04:59:01.937025 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:01.937027 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:01.937059 (+ 32us) server_negotiation.cc:407] Connection header received
0924 04:59:01.937116 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:01.937117 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:01.937136 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:01.937180 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:59:01.937289 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:01.937790 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:01.938497 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:01.938685 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:01.938735 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:01.938880 (+ 145us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:59:01.938913 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:59:01.938952 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:01.939002 (+ 50us) server_negotiation.cc:299] Negotiation successful
0924 04:59:01.939022 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 04:59:01.939117 2662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:01.936884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36967 (local address 127.0.98.2:43037)
0924 04:59:01.937009 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:01.937019 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:01.937073 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:01.937186 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:01.937187 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:01.937188 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 04:59:01.937241 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:01.937242 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:01.937854 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:01.937854 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:01.938429 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:01.938431 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:01.938741 (+ 310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:01.938742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:01.938799 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:01.938810 (+ 11us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 04:59:01.938961 (+ 151us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 04:59:01.938963 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 04:59:01.939001 (+ 38us) client_negotiation.cc:240] Negotiation successful
0924 04:59:01.939044 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":68,"threads_started":1}
I20260924 04:59:01.941073 2278 raft_consensus.cc:1261] T 9290b43630754f14abfdda9b1d780d27 P 2a10411e514d42a6b1e9fa6d6347eb83 [term 1 FOLLOWER]: Refusing update from remote peer 2e670717494940d193cc73c7ec81a78b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:59:01.941150 2558 raft_consensus.cc:1261] T 9290b43630754f14abfdda9b1d780d27 P ff5ba82680c5444aa7f89f73e77c4504 [term 1 FOLLOWER]: Refusing update from remote peer 2e670717494940d193cc73c7ec81a78b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:59:01.941401 2634 consensus_queue.cc:1035] T 9290b43630754f14abfdda9b1d780d27 P 2e670717494940d193cc73c7ec81a78b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff5ba82680c5444aa7f89f73e77c4504" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:59:01.941522 2630 consensus_queue.cc:1035] T 9290b43630754f14abfdda9b1d780d27 P 2e670717494940d193cc73c7ec81a78b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a10411e514d42a6b1e9fa6d6347eb83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:59:01.943179 2651 mvcc.cc:205] Tried to move back new op lower bound from 7332765458189103104 to 7332765457257185280. Current Snapshot: MvccSnapshot[applied={T|T < 7332765458189103104}]
I20260924 04:59:01.950054 2621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:01.947521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55185 (local address 127.0.98.2:36443)
0924 04:59:01.947555 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:01.947557 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:01.947588 (+ 31us) server_negotiation.cc:407] Connection header received
0924 04:59:01.947670 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:01.947671 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:01.947681 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:01.947725 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:59:01.947847 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:01.948560 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:01.949366 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:01.949641 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:01.949699 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:01.949846 (+ 147us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:59:01.949877 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:59:01.949925 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:01.949973 (+ 48us) server_negotiation.cc:299] Negotiation successful
0924 04:59:01.949990 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 04:59:01.950084 2610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:01.947430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36443 (local address 127.0.98.3:55185)
0924 04:59:01.947540 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:01.947550 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:01.947601 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:01.947731 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:01.947732 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:01.947733 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 04:59:01.947787 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:01.947789 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:01.948636 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:01.948636 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:01.949294 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:01.949297 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:01.949704 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:01.949705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:01.949769 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:01.949782 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 04:59:01.949931 (+ 149us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 04:59:01.949933 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 04:59:01.949972 (+ 39us) client_negotiation.cc:240] Negotiation successful
0924 04:59:01.950009 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260924 04:59:01.959113 2120 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59626:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:59:01.959208 2120 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59626:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:59:01.959832 2120 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cb927924e0fe40f185c2f20be51a64f0: Sending DeleteTablet for 3 replicas of tablet 9290b43630754f14abfdda9b1d780d27
I20260924 04:59:01.960098 2398 tablet_service.cc:1514] Processing DeleteTablet for tablet 9290b43630754f14abfdda9b1d780d27 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:59:01 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43346
I20260924 04:59:01.960220 2258 tablet_service.cc:1514] Processing DeleteTablet for tablet 9290b43630754f14abfdda9b1d780d27 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:59:01 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50284
I20260924 04:59:01.960429 2538 tablet_service.cc:1514] Processing DeleteTablet for tablet 9290b43630754f14abfdda9b1d780d27 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:59:01 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44996
I20260924 04:59:01.960700 2671 tablet_replica.cc:330] T 9290b43630754f14abfdda9b1d780d27 P 2a10411e514d42a6b1e9fa6d6347eb83: stopping tablet replica
I20260924 04:59:01.960668 2120 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59626:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:59:01.960820 2120 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:59:01.960994 2671 raft_consensus.cc:2229] T 9290b43630754f14abfdda9b1d780d27 P 2a10411e514d42a6b1e9fa6d6347eb83 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:59:01.961108 2671 raft_consensus.cc:2258] T 9290b43630754f14abfdda9b1d780d27 P 2a10411e514d42a6b1e9fa6d6347eb83 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:59:01.961973 2258 tablet_service.cc:1467] Processing CreateTablet for tablet f101b1258f404c15b1830d5d8e584282 (DEFAULT_TABLE table=test-table [id=e1c7b646560b486fa13f3fe828095316]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:59:01.961990 2538 tablet_service.cc:1467] Processing CreateTablet for tablet f101b1258f404c15b1830d5d8e584282 (DEFAULT_TABLE table=test-table [id=e1c7b646560b486fa13f3fe828095316]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:59:01.962087 2258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f101b1258f404c15b1830d5d8e584282. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:59:01.962086 2538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f101b1258f404c15b1830d5d8e584282. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:59:01.963354 2624 tablet_bootstrap.cc:492] T f101b1258f404c15b1830d5d8e584282 P ff5ba82680c5444aa7f89f73e77c4504: Bootstrap starting.
I20260924 04:59:01.963653 2624 tablet_bootstrap.cc:654] T f101b1258f404c15b1830d5d8e584282 P ff5ba82680c5444aa7f89f73e77c4504: Neither blocks nor log segments found. Creating new log.
I20260924 04:59:01.964052 2626 tablet_bootstrap.cc:492] T f101b1258f404c15b1830d5d8e584282 P 2a10411e514d42a6b1e9fa6d6347eb83: Bootstrap starting.
I20260924 04:59:01.964372 2626 tablet_bootstrap.cc:654] T f101b1258f404c15b1830d5d8e584282 P 2a10411e514d42a6b1e9fa6d6347eb83: Neither blocks nor log segments found. Creating new log.
I20260924 04:59:01.965078 2398 tablet_service.cc:1467] Processing CreateTablet for tablet f101b1258f404c15b1830d5d8e584282 (DEFAULT_TABLE table=test-table [id=e1c7b646560b486fa13f3fe828095316]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:59:01.965183 2398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f101b1258f404c15b1830d5d8e584282. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:59:01.965312 2626 tablet_bootstrap.cc:492] T f101b1258f404c15b1830d5d8e584282 P 2a10411e514d42a6b1e9fa6d6347eb83: No bootstrap required, opened a new log
I20260924 04:59:01.965351 2626 ts_tablet_manager.cc:1397] T f101b1258f404c15b1830d5d8e584282 P 2a10411e514d42a6b1e9fa6d6347eb83: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:59:01.965497 2626 raft_consensus.cc:348] T f101b1258f404c15b1830d5d8e584282 P 2a10411e514d42a6b1e9fa6d6347eb83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a10411e514d42a6b1e9fa6d6347eb83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } } peers { permanent_uuid: "2e670717494940d193cc73c7ec81a78b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36443 } } peers { permanent_uuid: "ff5ba82680c5444aa7f89f73e77c4504" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36967 } }
I20260924 04:59:01.965564 2626 raft_consensus.cc:374] T f101b1258f404c15b1830d5d8e584282 P 2a10411e514d42a6b1e9fa6d6347eb83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:59:01.965579 2626 raft_consensus.cc:729] T f101b1258f404c15b1830d5d8e584282 P 2a10411e514d42a6b1e9fa6d6347eb83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a10411e514d42a6b1e9fa6d6347eb83, State: Initialized, Role: FOLLOWER
I20260924 04:59:01.965632 2626 consensus_queue.cc:260] T f101b1258f404c15b1830d5d8e584282 P 2a10411e514d42a6b1e9fa6d6347eb83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a10411e514d42a6b1e9fa6d6347eb83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } } peers { permanent_uuid: "2e670717494940d193cc73c7ec81a78b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36443 } } peers { permanent_uuid: "ff5ba82680c5444aa7f89f73e77c4504" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36967 } }
I20260924 04:59:01.965704 2626 ts_tablet_manager.cc:1428] T f101b1258f404c15b1830d5d8e584282 P 2a10411e514d42a6b1e9fa6d6347eb83: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260924 04:59:01.966351 2624 tablet_bootstrap.cc:492] T f101b1258f404c15b1830d5d8e584282 P ff5ba82680c5444aa7f89f73e77c4504: No bootstrap required, opened a new log
I20260924 04:59:01.966391 2624 ts_tablet_manager.cc:1397] T f101b1258f404c15b1830d5d8e584282 P ff5ba82680c5444aa7f89f73e77c4504: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260924 04:59:01.966511 2625 tablet_bootstrap.cc:492] T f101b1258f404c15b1830d5d8e584282 P 2e670717494940d193cc73c7ec81a78b: Bootstrap starting.
I20260924 04:59:01.966521 2624 raft_consensus.cc:348] T f101b1258f404c15b1830d5d8e584282 P ff5ba82680c5444aa7f89f73e77c4504 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a10411e514d42a6b1e9fa6d6347eb83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } } peers { permanent_uuid: "2e670717494940d193cc73c7ec81a78b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36443 } } peers { permanent_uuid: "ff5ba82680c5444aa7f89f73e77c4504" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36967 } }
I20260924 04:59:01.966578 2624 raft_consensus.cc:374] T f101b1258f404c15b1830d5d8e584282 P ff5ba82680c5444aa7f89f73e77c4504 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:59:01.966598 2624 raft_consensus.cc:729] T f101b1258f404c15b1830d5d8e584282 P ff5ba82680c5444aa7f89f73e77c4504 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff5ba82680c5444aa7f89f73e77c4504, State: Initialized, Role: FOLLOWER
I20260924 04:59:01.966636 2624 consensus_queue.cc:260] T f101b1258f404c15b1830d5d8e584282 P ff5ba82680c5444aa7f89f73e77c4504 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a10411e514d42a6b1e9fa6d6347eb83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } } peers { permanent_uuid: "2e670717494940d193cc73c7ec81a78b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36443 } } peers { permanent_uuid: "ff5ba82680c5444aa7f89f73e77c4504" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36967 } }
I20260924 04:59:01.966723 2624 ts_tablet_manager.cc:1428] T f101b1258f404c15b1830d5d8e584282 P ff5ba82680c5444aa7f89f73e77c4504: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:01.966874 2625 tablet_bootstrap.cc:654] T f101b1258f404c15b1830d5d8e584282 P 2e670717494940d193cc73c7ec81a78b: Neither blocks nor log segments found. Creating new log.
I20260924 04:59:01.967449 2671 ts_tablet_manager.cc:1905] T 9290b43630754f14abfdda9b1d780d27 P 2a10411e514d42a6b1e9fa6d6347eb83: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:59:01.968811 2672 tablet_replica.cc:330] T 9290b43630754f14abfdda9b1d780d27 P ff5ba82680c5444aa7f89f73e77c4504: stopping tablet replica
I20260924 04:59:01.968927 2672 raft_consensus.cc:2229] T 9290b43630754f14abfdda9b1d780d27 P ff5ba82680c5444aa7f89f73e77c4504 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:59:01.969012 2670 tablet_replica.cc:330] T 9290b43630754f14abfdda9b1d780d27 P 2e670717494940d193cc73c7ec81a78b: stopping tablet replica
I20260924 04:59:01.969014 2671 ts_tablet_manager.cc:1918] T 9290b43630754f14abfdda9b1d780d27 P 2a10411e514d42a6b1e9fa6d6347eb83: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 04:59:01.969070 2671 log.cc:1196] T 9290b43630754f14abfdda9b1d780d27 P 2a10411e514d42a6b1e9fa6d6347eb83: Deleting WAL directory at /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225929848825-392-0/minicluster-data/ts-0/wal/wals/9290b43630754f14abfdda9b1d780d27
I20260924 04:59:01.969187 2670 raft_consensus.cc:2229] T 9290b43630754f14abfdda9b1d780d27 P 2e670717494940d193cc73c7ec81a78b [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:59:01.969348 2671 ts_tablet_manager.cc:1939] T 9290b43630754f14abfdda9b1d780d27 P 2a10411e514d42a6b1e9fa6d6347eb83: Deleting consensus metadata
I20260924 04:59:01.969347 2670 raft_consensus.cc:2258] T 9290b43630754f14abfdda9b1d780d27 P 2e670717494940d193cc73c7ec81a78b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:59:01.969486 2672 raft_consensus.cc:2258] T 9290b43630754f14abfdda9b1d780d27 P ff5ba82680c5444aa7f89f73e77c4504 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:59:01.969842 2105 catalog_manager.cc:4841] TS 2a10411e514d42a6b1e9fa6d6347eb83 (127.0.98.1:38697): tablet 9290b43630754f14abfdda9b1d780d27 (table test-table [id=fdfd922e0c7b476787f5d9826af5a3c4]) successfully deleted
I20260924 04:59:01.970042 2672 ts_tablet_manager.cc:1905] T 9290b43630754f14abfdda9b1d780d27 P ff5ba82680c5444aa7f89f73e77c4504: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:59:01.971387 2670 ts_tablet_manager.cc:1905] T 9290b43630754f14abfdda9b1d780d27 P 2e670717494940d193cc73c7ec81a78b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:59:01.971561 2672 ts_tablet_manager.cc:1918] T 9290b43630754f14abfdda9b1d780d27 P ff5ba82680c5444aa7f89f73e77c4504: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 04:59:01.971663 2672 log.cc:1196] T 9290b43630754f14abfdda9b1d780d27 P ff5ba82680c5444aa7f89f73e77c4504: Deleting WAL directory at /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225929848825-392-0/minicluster-data/ts-2/wal/wals/9290b43630754f14abfdda9b1d780d27
I20260924 04:59:01.971946 2672 ts_tablet_manager.cc:1939] T 9290b43630754f14abfdda9b1d780d27 P ff5ba82680c5444aa7f89f73e77c4504: Deleting consensus metadata
I20260924 04:59:01.972370 2107 catalog_manager.cc:4841] TS ff5ba82680c5444aa7f89f73e77c4504 (127.0.98.3:36967): tablet 9290b43630754f14abfdda9b1d780d27 (table test-table [id=fdfd922e0c7b476787f5d9826af5a3c4]) successfully deleted
I20260924 04:59:01.972779 2670 ts_tablet_manager.cc:1918] T 9290b43630754f14abfdda9b1d780d27 P 2e670717494940d193cc73c7ec81a78b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 04:59:01.972836 2670 log.cc:1196] T 9290b43630754f14abfdda9b1d780d27 P 2e670717494940d193cc73c7ec81a78b: Deleting WAL directory at /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225929848825-392-0/minicluster-data/ts-1/wal/wals/9290b43630754f14abfdda9b1d780d27
I20260924 04:59:01.973021 2670 ts_tablet_manager.cc:1939] T 9290b43630754f14abfdda9b1d780d27 P 2e670717494940d193cc73c7ec81a78b: Deleting consensus metadata
I20260924 04:59:01.973392 2106 catalog_manager.cc:4841] TS 2e670717494940d193cc73c7ec81a78b (127.0.98.2:36443): tablet 9290b43630754f14abfdda9b1d780d27 (table test-table [id=fdfd922e0c7b476787f5d9826af5a3c4]) successfully deleted
I20260924 04:59:01.973744 2625 tablet_bootstrap.cc:492] T f101b1258f404c15b1830d5d8e584282 P 2e670717494940d193cc73c7ec81a78b: No bootstrap required, opened a new log
I20260924 04:59:01.973802 2625 ts_tablet_manager.cc:1397] T f101b1258f404c15b1830d5d8e584282 P 2e670717494940d193cc73c7ec81a78b: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260924 04:59:01.973932 2625 raft_consensus.cc:348] T f101b1258f404c15b1830d5d8e584282 P 2e670717494940d193cc73c7ec81a78b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a10411e514d42a6b1e9fa6d6347eb83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } } peers { permanent_uuid: "2e670717494940d193cc73c7ec81a78b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36443 } } peers { permanent_uuid: "ff5ba82680c5444aa7f89f73e77c4504" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36967 } }
I20260924 04:59:01.973994 2625 raft_consensus.cc:374] T f101b1258f404c15b1830d5d8e584282 P 2e670717494940d193cc73c7ec81a78b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:59:01.974011 2625 raft_consensus.cc:729] T f101b1258f404c15b1830d5d8e584282 P 2e670717494940d193cc73c7ec81a78b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e670717494940d193cc73c7ec81a78b, State: Initialized, Role: FOLLOWER
I20260924 04:59:01.974050 2625 consensus_queue.cc:260] T f101b1258f404c15b1830d5d8e584282 P 2e670717494940d193cc73c7ec81a78b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a10411e514d42a6b1e9fa6d6347eb83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } } peers { permanent_uuid: "2e670717494940d193cc73c7ec81a78b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36443 } } peers { permanent_uuid: "ff5ba82680c5444aa7f89f73e77c4504" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36967 } }
I20260924 04:59:01.974133 2625 ts_tablet_manager.cc:1428] T f101b1258f404c15b1830d5d8e584282 P 2e670717494940d193cc73c7ec81a78b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:02.166522 2666 raft_consensus.cc:482] T f101b1258f404c15b1830d5d8e584282 P ff5ba82680c5444aa7f89f73e77c4504 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:59:02.166615 2666 raft_consensus.cc:504] T f101b1258f404c15b1830d5d8e584282 P ff5ba82680c5444aa7f89f73e77c4504 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a10411e514d42a6b1e9fa6d6347eb83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } } peers { permanent_uuid: "2e670717494940d193cc73c7ec81a78b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36443 } } peers { permanent_uuid: "ff5ba82680c5444aa7f89f73e77c4504" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36967 } }
I20260924 04:59:02.166754 2666 leader_election.cc:288] T f101b1258f404c15b1830d5d8e584282 P ff5ba82680c5444aa7f89f73e77c4504 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a10411e514d42a6b1e9fa6d6347eb83 (127.0.98.1:38697), 2e670717494940d193cc73c7ec81a78b (127.0.98.2:36443)
I20260924 04:59:02.167026 2278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f101b1258f404c15b1830d5d8e584282" candidate_uuid: "ff5ba82680c5444aa7f89f73e77c4504" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a10411e514d42a6b1e9fa6d6347eb83" is_pre_election: true
I20260924 04:59:02.167006 2418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f101b1258f404c15b1830d5d8e584282" candidate_uuid: "ff5ba82680c5444aa7f89f73e77c4504" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e670717494940d193cc73c7ec81a78b" is_pre_election: true
I20260924 04:59:02.167114 2278 raft_consensus.cc:2454] T f101b1258f404c15b1830d5d8e584282 P 2a10411e514d42a6b1e9fa6d6347eb83 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff5ba82680c5444aa7f89f73e77c4504 in term 0.
I20260924 04:59:02.167114 2418 raft_consensus.cc:2454] T f101b1258f404c15b1830d5d8e584282 P 2e670717494940d193cc73c7ec81a78b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff5ba82680c5444aa7f89f73e77c4504 in term 0.
I20260924 04:59:02.167268 2489 leader_election.cc:302] T f101b1258f404c15b1830d5d8e584282 P ff5ba82680c5444aa7f89f73e77c4504 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a10411e514d42a6b1e9fa6d6347eb83, ff5ba82680c5444aa7f89f73e77c4504; no voters:
I20260924 04:59:02.167366 2666 raft_consensus.cc:2781] T f101b1258f404c15b1830d5d8e584282 P ff5ba82680c5444aa7f89f73e77c4504 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:59:02.167419 2666 raft_consensus.cc:482] T f101b1258f404c15b1830d5d8e584282 P ff5ba82680c5444aa7f89f73e77c4504 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:59:02.167443 2666 raft_consensus.cc:3037] T f101b1258f404c15b1830d5d8e584282 P ff5ba82680c5444aa7f89f73e77c4504 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:59:02.168215 2666 raft_consensus.cc:504] T f101b1258f404c15b1830d5d8e584282 P ff5ba82680c5444aa7f89f73e77c4504 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a10411e514d42a6b1e9fa6d6347eb83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } } peers { permanent_uuid: "2e670717494940d193cc73c7ec81a78b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36443 } } peers { permanent_uuid: "ff5ba82680c5444aa7f89f73e77c4504" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36967 } }
I20260924 04:59:02.168336 2666 leader_election.cc:288] T f101b1258f404c15b1830d5d8e584282 P ff5ba82680c5444aa7f89f73e77c4504 [CANDIDATE]: Term 1 election: Requested vote from peers 2a10411e514d42a6b1e9fa6d6347eb83 (127.0.98.1:38697), 2e670717494940d193cc73c7ec81a78b (127.0.98.2:36443)
I20260924 04:59:02.168473 2278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f101b1258f404c15b1830d5d8e584282" candidate_uuid: "ff5ba82680c5444aa7f89f73e77c4504" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a10411e514d42a6b1e9fa6d6347eb83"
I20260924 04:59:02.168497 2418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f101b1258f404c15b1830d5d8e584282" candidate_uuid: "ff5ba82680c5444aa7f89f73e77c4504" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e670717494940d193cc73c7ec81a78b"
I20260924 04:59:02.168538 2278 raft_consensus.cc:3037] T f101b1258f404c15b1830d5d8e584282 P 2a10411e514d42a6b1e9fa6d6347eb83 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:59:02.168556 2418 raft_consensus.cc:3037] T f101b1258f404c15b1830d5d8e584282 P 2e670717494940d193cc73c7ec81a78b [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:59:02.169078 2278 raft_consensus.cc:2454] T f101b1258f404c15b1830d5d8e584282 P 2a10411e514d42a6b1e9fa6d6347eb83 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff5ba82680c5444aa7f89f73e77c4504 in term 1.
I20260924 04:59:02.169162 2418 raft_consensus.cc:2454] T f101b1258f404c15b1830d5d8e584282 P 2e670717494940d193cc73c7ec81a78b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff5ba82680c5444aa7f89f73e77c4504 in term 1.
I20260924 04:59:02.169224 2489 leader_election.cc:302] T f101b1258f404c15b1830d5d8e584282 P ff5ba82680c5444aa7f89f73e77c4504 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a10411e514d42a6b1e9fa6d6347eb83, ff5ba82680c5444aa7f89f73e77c4504; no voters:
I20260924 04:59:02.169324 2666 raft_consensus.cc:2781] T f101b1258f404c15b1830d5d8e584282 P ff5ba82680c5444aa7f89f73e77c4504 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:59:02.169392 2666 raft_consensus.cc:686] T f101b1258f404c15b1830d5d8e584282 P ff5ba82680c5444aa7f89f73e77c4504 [term 1 LEADER]: Becoming Leader. State: Replica: ff5ba82680c5444aa7f89f73e77c4504, State: Running, Role: LEADER
I20260924 04:59:02.169457 2666 consensus_queue.cc:237] T f101b1258f404c15b1830d5d8e584282 P ff5ba82680c5444aa7f89f73e77c4504 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a10411e514d42a6b1e9fa6d6347eb83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } } peers { permanent_uuid: "2e670717494940d193cc73c7ec81a78b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36443 } } peers { permanent_uuid: "ff5ba82680c5444aa7f89f73e77c4504" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36967 } }
I20260924 04:59:02.169981 2121 catalog_manager.cc:5495] T f101b1258f404c15b1830d5d8e584282 P ff5ba82680c5444aa7f89f73e77c4504 reported cstate change: term changed from 0 to 1, leader changed from <none> to ff5ba82680c5444aa7f89f73e77c4504 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "ff5ba82680c5444aa7f89f73e77c4504" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a10411e514d42a6b1e9fa6d6347eb83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e670717494940d193cc73c7ec81a78b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff5ba82680c5444aa7f89f73e77c4504" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36967 } health_report { overall_health: HEALTHY } } }
I20260924 04:59:02.195657 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:02.193704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45010 (local address 127.0.98.3:36967)
0924 04:59:02.193744 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:02.193747 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:02.193756 (+ 9us) server_negotiation.cc:407] Connection header received
0924 04:59:02.193795 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:02.193796 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:02.193808 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:02.193848 (+ 40us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:59:02.193941 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:02.194390 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:02.195163 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:02.195352 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:02.195410 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:02.195497 (+ 87us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:59:02.195523 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:59:02.195553 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:02.195584 (+ 31us) server_negotiation.cc:299] Negotiation successful
0924 04:59:02.195598 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 04:59:02.199307 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:02.197623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54397 (local address 127.0.98.3:36967)
0924 04:59:02.197662 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:02.197664 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:02.197670 (+ 6us) server_negotiation.cc:407] Connection header received
0924 04:59:02.197684 (+ 14us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:02.197685 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:02.197691 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:02.197726 (+ 35us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:59:02.197800 (+ 74us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:02.198209 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:02.198840 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:02.198997 (+ 157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:02.199036 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:02.199139 (+ 103us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:59:02.199159 (+ 20us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:59:02.199189 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:02.199225 (+ 36us) server_negotiation.cc:299] Negotiation successful
0924 04:59:02.199237 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":25}
I20260924 04:59:02.199476 2610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:02.197492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36967 (local address 127.0.98.3:54397)
0924 04:59:02.197534 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:02.197544 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:02.197577 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:02.197729 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:02.197730 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:02.197731 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 04:59:02.197760 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:02.197761 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:02.198257 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:02.198257 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:02.198777 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:02.198779 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:02.199046 (+ 267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:02.199047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:02.199087 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:02.199091 (+ 4us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 04:59:02.199189 (+ 98us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 04:59:02.199190 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 04:59:02.199225 (+ 35us) client_negotiation.cc:240] Negotiation successful
0924 04:59:02.199259 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260924 04:59:02.201144 2278 raft_consensus.cc:1261] T f101b1258f404c15b1830d5d8e584282 P 2a10411e514d42a6b1e9fa6d6347eb83 [term 1 FOLLOWER]: Refusing update from remote peer ff5ba82680c5444aa7f89f73e77c4504: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:59:02.201170 2418 raft_consensus.cc:1261] T f101b1258f404c15b1830d5d8e584282 P 2e670717494940d193cc73c7ec81a78b [term 1 FOLLOWER]: Refusing update from remote peer ff5ba82680c5444aa7f89f73e77c4504: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:59:02.201342 2666 consensus_queue.cc:1035] T f101b1258f404c15b1830d5d8e584282 P ff5ba82680c5444aa7f89f73e77c4504 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a10411e514d42a6b1e9fa6d6347eb83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:59:02.201457 2631 consensus_queue.cc:1035] T f101b1258f404c15b1830d5d8e584282 P ff5ba82680c5444aa7f89f73e77c4504 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e670717494940d193cc73c7ec81a78b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 04:59:02.205366 2651 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 04:59:03.723457 2121 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59626:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:59:03.723564 2121 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59626:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:59:03.724395 2121 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cb927924e0fe40f185c2f20be51a64f0: Sending DeleteTablet for 3 replicas of tablet f101b1258f404c15b1830d5d8e584282
I20260924 04:59:03.724660 2258 tablet_service.cc:1514] Processing DeleteTablet for tablet f101b1258f404c15b1830d5d8e584282 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:59:03 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50284
I20260924 04:59:03.724668 2398 tablet_service.cc:1514] Processing DeleteTablet for tablet f101b1258f404c15b1830d5d8e584282 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:59:03 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43346
I20260924 04:59:03.724705 2537 tablet_service.cc:1514] Processing DeleteTablet for tablet f101b1258f404c15b1830d5d8e584282 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:59:03 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44996
I20260924 04:59:03.724874 2701 tablet_replica.cc:330] T f101b1258f404c15b1830d5d8e584282 P 2a10411e514d42a6b1e9fa6d6347eb83: stopping tablet replica
I20260924 04:59:03.724924 2701 raft_consensus.cc:2229] T f101b1258f404c15b1830d5d8e584282 P 2a10411e514d42a6b1e9fa6d6347eb83 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:59:03.724959 2701 raft_consensus.cc:2258] T f101b1258f404c15b1830d5d8e584282 P 2a10411e514d42a6b1e9fa6d6347eb83 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:59:03.725116 2703 tablet_replica.cc:330] T f101b1258f404c15b1830d5d8e584282 P 2e670717494940d193cc73c7ec81a78b: stopping tablet replica
I20260924 04:59:03.725164 2703 raft_consensus.cc:2229] T f101b1258f404c15b1830d5d8e584282 P 2e670717494940d193cc73c7ec81a78b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:59:03.725194 2703 raft_consensus.cc:2258] T f101b1258f404c15b1830d5d8e584282 P 2e670717494940d193cc73c7ec81a78b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:59:03.725210 2701 ts_tablet_manager.cc:1905] T f101b1258f404c15b1830d5d8e584282 P 2a10411e514d42a6b1e9fa6d6347eb83: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:59:03.725391 2703 ts_tablet_manager.cc:1905] T f101b1258f404c15b1830d5d8e584282 P 2e670717494940d193cc73c7ec81a78b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:59:03.726799 2702 tablet_replica.cc:330] T f101b1258f404c15b1830d5d8e584282 P ff5ba82680c5444aa7f89f73e77c4504: stopping tablet replica
I20260924 04:59:03.726919 2702 raft_consensus.cc:2229] T f101b1258f404c15b1830d5d8e584282 P ff5ba82680c5444aa7f89f73e77c4504 [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:59:03.727020 2702 raft_consensus.cc:2258] T f101b1258f404c15b1830d5d8e584282 P ff5ba82680c5444aa7f89f73e77c4504 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:59:03.727015 2701 ts_tablet_manager.cc:1918] T f101b1258f404c15b1830d5d8e584282 P 2a10411e514d42a6b1e9fa6d6347eb83: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 04:59:03.727061 2701 log.cc:1196] T f101b1258f404c15b1830d5d8e584282 P 2a10411e514d42a6b1e9fa6d6347eb83: Deleting WAL directory at /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225929848825-392-0/minicluster-data/ts-0/wal/wals/f101b1258f404c15b1830d5d8e584282
I20260924 04:59:03.727278 2701 ts_tablet_manager.cc:1939] T f101b1258f404c15b1830d5d8e584282 P 2a10411e514d42a6b1e9fa6d6347eb83: Deleting consensus metadata
I20260924 04:59:03.727311 2702 ts_tablet_manager.cc:1905] T f101b1258f404c15b1830d5d8e584282 P ff5ba82680c5444aa7f89f73e77c4504: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:59:03.727622 2105 catalog_manager.cc:4841] TS 2a10411e514d42a6b1e9fa6d6347eb83 (127.0.98.1:38697): tablet f101b1258f404c15b1830d5d8e584282 (table test-table [id=e1c7b646560b486fa13f3fe828095316]) successfully deleted
I20260924 04:59:03.728708 2703 ts_tablet_manager.cc:1918] T f101b1258f404c15b1830d5d8e584282 P 2e670717494940d193cc73c7ec81a78b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 04:59:03.728772 2703 log.cc:1196] T f101b1258f404c15b1830d5d8e584282 P 2e670717494940d193cc73c7ec81a78b: Deleting WAL directory at /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225929848825-392-0/minicluster-data/ts-1/wal/wals/f101b1258f404c15b1830d5d8e584282
I20260924 04:59:03.728827 2702 ts_tablet_manager.cc:1918] T f101b1258f404c15b1830d5d8e584282 P ff5ba82680c5444aa7f89f73e77c4504: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 04:59:03.728876 2702 log.cc:1196] T f101b1258f404c15b1830d5d8e584282 P ff5ba82680c5444aa7f89f73e77c4504: Deleting WAL directory at /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225929848825-392-0/minicluster-data/ts-2/wal/wals/f101b1258f404c15b1830d5d8e584282
I20260924 04:59:03.728984 2703 ts_tablet_manager.cc:1939] T f101b1258f404c15b1830d5d8e584282 P 2e670717494940d193cc73c7ec81a78b: Deleting consensus metadata
I20260924 04:59:03.729060 2702 ts_tablet_manager.cc:1939] T f101b1258f404c15b1830d5d8e584282 P ff5ba82680c5444aa7f89f73e77c4504: Deleting consensus metadata
I20260924 04:59:03.729467 2106 catalog_manager.cc:4841] TS 2e670717494940d193cc73c7ec81a78b (127.0.98.2:36443): tablet f101b1258f404c15b1830d5d8e584282 (table test-table [id=e1c7b646560b486fa13f3fe828095316]) successfully deleted
I20260924 04:59:03.729616 2107 catalog_manager.cc:4841] TS ff5ba82680c5444aa7f89f73e77c4504 (127.0.98.3:36967): tablet f101b1258f404c15b1830d5d8e584282 (table test-table [id=e1c7b646560b486fa13f3fe828095316]) successfully deleted
Sep 24 04:59:03 dist-test-slave-70st krb5kdc[2076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225943, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 04:59:03 dist-test-slave-70st krb5kdc[2076](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225943, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:59:03.757671 2711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:03.742606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59648 (local address 127.0.98.62:42637)
0924 04:59:03.742748 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:03.742750 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:03.742783 (+ 33us) server_negotiation.cc:407] Connection header received
0924 04:59:03.742810 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:03.742811 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:03.742821 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:03.742877 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:03.743425 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:03.744016 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:03.744823 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:03.745060 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:03.745121 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:03.755361 (+ 10240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:03.755363 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:03.755363 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:59:03.755373 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:03.756613 (+ 1240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:03.756918 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:03.756919 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:03.756920 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:03.756986 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:03.757149 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:03.757150 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:03.757150 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:03.757263 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:03.757355 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:03.757476 (+ 121us) server_negotiation.cc:299] Negotiation successful
0924 04:59:03.757563 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":67,"threads_started":1}
I20260924 04:59:03.759027 2121 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59648:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:59:03.759181 2121 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:59:03.760492 2258 tablet_service.cc:1467] Processing CreateTablet for tablet a800aab720df4846afbf2fb20b2226e7 (DEFAULT_TABLE table=test-table [id=caa1b493daa34f8d95af9740c368bee5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:59:03.760514 2398 tablet_service.cc:1467] Processing CreateTablet for tablet a800aab720df4846afbf2fb20b2226e7 (DEFAULT_TABLE table=test-table [id=caa1b493daa34f8d95af9740c368bee5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:59:03.760541 2537 tablet_service.cc:1467] Processing CreateTablet for tablet a800aab720df4846afbf2fb20b2226e7 (DEFAULT_TABLE table=test-table [id=caa1b493daa34f8d95af9740c368bee5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:59:03.760643 2398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a800aab720df4846afbf2fb20b2226e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:59:03.760640 2258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a800aab720df4846afbf2fb20b2226e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:59:03.760660 2537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a800aab720df4846afbf2fb20b2226e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:59:03.761840 2712 tablet_bootstrap.cc:492] T a800aab720df4846afbf2fb20b2226e7 P ff5ba82680c5444aa7f89f73e77c4504: Bootstrap starting.
I20260924 04:59:03.762157 2714 tablet_bootstrap.cc:492] T a800aab720df4846afbf2fb20b2226e7 P 2a10411e514d42a6b1e9fa6d6347eb83: Bootstrap starting.
I20260924 04:59:03.762269 2712 tablet_bootstrap.cc:654] T a800aab720df4846afbf2fb20b2226e7 P ff5ba82680c5444aa7f89f73e77c4504: Neither blocks nor log segments found. Creating new log.
I20260924 04:59:03.762494 2714 tablet_bootstrap.cc:654] T a800aab720df4846afbf2fb20b2226e7 P 2a10411e514d42a6b1e9fa6d6347eb83: Neither blocks nor log segments found. Creating new log.
I20260924 04:59:03.762804 2713 tablet_bootstrap.cc:492] T a800aab720df4846afbf2fb20b2226e7 P 2e670717494940d193cc73c7ec81a78b: Bootstrap starting.
I20260924 04:59:03.762938 2712 tablet_bootstrap.cc:492] T a800aab720df4846afbf2fb20b2226e7 P ff5ba82680c5444aa7f89f73e77c4504: No bootstrap required, opened a new log
I20260924 04:59:03.763034 2714 tablet_bootstrap.cc:492] T a800aab720df4846afbf2fb20b2226e7 P 2a10411e514d42a6b1e9fa6d6347eb83: No bootstrap required, opened a new log
I20260924 04:59:03.763065 2712 ts_tablet_manager.cc:1397] T a800aab720df4846afbf2fb20b2226e7 P ff5ba82680c5444aa7f89f73e77c4504: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:59:03.763106 2714 ts_tablet_manager.cc:1397] T a800aab720df4846afbf2fb20b2226e7 P 2a10411e514d42a6b1e9fa6d6347eb83: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:59:03.763234 2713 tablet_bootstrap.cc:654] T a800aab720df4846afbf2fb20b2226e7 P 2e670717494940d193cc73c7ec81a78b: Neither blocks nor log segments found. Creating new log.
I20260924 04:59:03.763211 2712 raft_consensus.cc:348] T a800aab720df4846afbf2fb20b2226e7 P ff5ba82680c5444aa7f89f73e77c4504 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a10411e514d42a6b1e9fa6d6347eb83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } } peers { permanent_uuid: "2e670717494940d193cc73c7ec81a78b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36443 } } peers { permanent_uuid: "ff5ba82680c5444aa7f89f73e77c4504" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36967 } }
I20260924 04:59:03.763276 2712 raft_consensus.cc:374] T a800aab720df4846afbf2fb20b2226e7 P ff5ba82680c5444aa7f89f73e77c4504 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:59:03.763252 2714 raft_consensus.cc:348] T a800aab720df4846afbf2fb20b2226e7 P 2a10411e514d42a6b1e9fa6d6347eb83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a10411e514d42a6b1e9fa6d6347eb83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } } peers { permanent_uuid: "2e670717494940d193cc73c7ec81a78b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36443 } } peers { permanent_uuid: "ff5ba82680c5444aa7f89f73e77c4504" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36967 } }
I20260924 04:59:03.763294 2712 raft_consensus.cc:729] T a800aab720df4846afbf2fb20b2226e7 P ff5ba82680c5444aa7f89f73e77c4504 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff5ba82680c5444aa7f89f73e77c4504, State: Initialized, Role: FOLLOWER
I20260924 04:59:03.763311 2714 raft_consensus.cc:374] T a800aab720df4846afbf2fb20b2226e7 P 2a10411e514d42a6b1e9fa6d6347eb83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:59:03.763355 2714 raft_consensus.cc:729] T a800aab720df4846afbf2fb20b2226e7 P 2a10411e514d42a6b1e9fa6d6347eb83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a10411e514d42a6b1e9fa6d6347eb83, State: Initialized, Role: FOLLOWER
I20260924 04:59:03.763355 2712 consensus_queue.cc:260] T a800aab720df4846afbf2fb20b2226e7 P ff5ba82680c5444aa7f89f73e77c4504 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a10411e514d42a6b1e9fa6d6347eb83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } } peers { permanent_uuid: "2e670717494940d193cc73c7ec81a78b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36443 } } peers { permanent_uuid: "ff5ba82680c5444aa7f89f73e77c4504" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36967 } }
I20260924 04:59:03.763432 2714 consensus_queue.cc:260] T a800aab720df4846afbf2fb20b2226e7 P 2a10411e514d42a6b1e9fa6d6347eb83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a10411e514d42a6b1e9fa6d6347eb83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } } peers { permanent_uuid: "2e670717494940d193cc73c7ec81a78b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36443 } } peers { permanent_uuid: "ff5ba82680c5444aa7f89f73e77c4504" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36967 } }
I20260924 04:59:03.763643 2714 ts_tablet_manager.cc:1428] T a800aab720df4846afbf2fb20b2226e7 P 2a10411e514d42a6b1e9fa6d6347eb83: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:03.763643 2712 ts_tablet_manager.cc:1428] T a800aab720df4846afbf2fb20b2226e7 P ff5ba82680c5444aa7f89f73e77c4504: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 04:59:03.763864 2713 tablet_bootstrap.cc:492] T a800aab720df4846afbf2fb20b2226e7 P 2e670717494940d193cc73c7ec81a78b: No bootstrap required, opened a new log
I20260924 04:59:03.763975 2713 ts_tablet_manager.cc:1397] T a800aab720df4846afbf2fb20b2226e7 P 2e670717494940d193cc73c7ec81a78b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:59:03.764253 2713 raft_consensus.cc:348] T a800aab720df4846afbf2fb20b2226e7 P 2e670717494940d193cc73c7ec81a78b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a10411e514d42a6b1e9fa6d6347eb83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } } peers { permanent_uuid: "2e670717494940d193cc73c7ec81a78b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36443 } } peers { permanent_uuid: "ff5ba82680c5444aa7f89f73e77c4504" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36967 } }
I20260924 04:59:03.764348 2713 raft_consensus.cc:374] T a800aab720df4846afbf2fb20b2226e7 P 2e670717494940d193cc73c7ec81a78b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:59:03.764369 2713 raft_consensus.cc:729] T a800aab720df4846afbf2fb20b2226e7 P 2e670717494940d193cc73c7ec81a78b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e670717494940d193cc73c7ec81a78b, State: Initialized, Role: FOLLOWER
I20260924 04:59:03.764408 2713 consensus_queue.cc:260] T a800aab720df4846afbf2fb20b2226e7 P 2e670717494940d193cc73c7ec81a78b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a10411e514d42a6b1e9fa6d6347eb83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } } peers { permanent_uuid: "2e670717494940d193cc73c7ec81a78b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36443 } } peers { permanent_uuid: "ff5ba82680c5444aa7f89f73e77c4504" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36967 } }
I20260924 04:59:03.764569 2713 ts_tablet_manager.cc:1428] T a800aab720df4846afbf2fb20b2226e7 P 2e670717494940d193cc73c7ec81a78b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:03.839865 2719 raft_consensus.cc:482] T a800aab720df4846afbf2fb20b2226e7 P 2e670717494940d193cc73c7ec81a78b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:59:03.839964 2719 raft_consensus.cc:504] T a800aab720df4846afbf2fb20b2226e7 P 2e670717494940d193cc73c7ec81a78b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a10411e514d42a6b1e9fa6d6347eb83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } } peers { permanent_uuid: "2e670717494940d193cc73c7ec81a78b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36443 } } peers { permanent_uuid: "ff5ba82680c5444aa7f89f73e77c4504" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36967 } }
I20260924 04:59:03.840119 2719 leader_election.cc:288] T a800aab720df4846afbf2fb20b2226e7 P 2e670717494940d193cc73c7ec81a78b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a10411e514d42a6b1e9fa6d6347eb83 (127.0.98.1:38697), ff5ba82680c5444aa7f89f73e77c4504 (127.0.98.3:36967)
I20260924 04:59:03.840355 2278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a800aab720df4846afbf2fb20b2226e7" candidate_uuid: "2e670717494940d193cc73c7ec81a78b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a10411e514d42a6b1e9fa6d6347eb83" is_pre_election: true
I20260924 04:59:03.840355 2558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a800aab720df4846afbf2fb20b2226e7" candidate_uuid: "2e670717494940d193cc73c7ec81a78b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5ba82680c5444aa7f89f73e77c4504" is_pre_election: true
I20260924 04:59:03.840456 2558 raft_consensus.cc:2454] T a800aab720df4846afbf2fb20b2226e7 P ff5ba82680c5444aa7f89f73e77c4504 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e670717494940d193cc73c7ec81a78b in term 0.
I20260924 04:59:03.840456 2278 raft_consensus.cc:2454] T a800aab720df4846afbf2fb20b2226e7 P 2a10411e514d42a6b1e9fa6d6347eb83 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e670717494940d193cc73c7ec81a78b in term 0.
I20260924 04:59:03.840616 2351 leader_election.cc:302] T a800aab720df4846afbf2fb20b2226e7 P 2e670717494940d193cc73c7ec81a78b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e670717494940d193cc73c7ec81a78b, ff5ba82680c5444aa7f89f73e77c4504; no voters:
I20260924 04:59:03.840700 2719 raft_consensus.cc:2781] T a800aab720df4846afbf2fb20b2226e7 P 2e670717494940d193cc73c7ec81a78b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:59:03.840739 2719 raft_consensus.cc:482] T a800aab720df4846afbf2fb20b2226e7 P 2e670717494940d193cc73c7ec81a78b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:59:03.840750 2719 raft_consensus.cc:3037] T a800aab720df4846afbf2fb20b2226e7 P 2e670717494940d193cc73c7ec81a78b [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:59:03.841544 2719 raft_consensus.cc:504] T a800aab720df4846afbf2fb20b2226e7 P 2e670717494940d193cc73c7ec81a78b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a10411e514d42a6b1e9fa6d6347eb83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } } peers { permanent_uuid: "2e670717494940d193cc73c7ec81a78b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36443 } } peers { permanent_uuid: "ff5ba82680c5444aa7f89f73e77c4504" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36967 } }
I20260924 04:59:03.841727 2719 leader_election.cc:288] T a800aab720df4846afbf2fb20b2226e7 P 2e670717494940d193cc73c7ec81a78b [CANDIDATE]: Term 1 election: Requested vote from peers 2a10411e514d42a6b1e9fa6d6347eb83 (127.0.98.1:38697), ff5ba82680c5444aa7f89f73e77c4504 (127.0.98.3:36967)
I20260924 04:59:03.841784 2558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a800aab720df4846afbf2fb20b2226e7" candidate_uuid: "2e670717494940d193cc73c7ec81a78b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5ba82680c5444aa7f89f73e77c4504"
I20260924 04:59:03.841866 2558 raft_consensus.cc:3037] T a800aab720df4846afbf2fb20b2226e7 P ff5ba82680c5444aa7f89f73e77c4504 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:59:03.841843 2278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a800aab720df4846afbf2fb20b2226e7" candidate_uuid: "2e670717494940d193cc73c7ec81a78b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a10411e514d42a6b1e9fa6d6347eb83"
I20260924 04:59:03.841900 2278 raft_consensus.cc:3037] T a800aab720df4846afbf2fb20b2226e7 P 2a10411e514d42a6b1e9fa6d6347eb83 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:59:03.842615 2278 raft_consensus.cc:2454] T a800aab720df4846afbf2fb20b2226e7 P 2a10411e514d42a6b1e9fa6d6347eb83 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e670717494940d193cc73c7ec81a78b in term 1.
I20260924 04:59:03.842657 2558 raft_consensus.cc:2454] T a800aab720df4846afbf2fb20b2226e7 P ff5ba82680c5444aa7f89f73e77c4504 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e670717494940d193cc73c7ec81a78b in term 1.
I20260924 04:59:03.842794 2349 leader_election.cc:302] T a800aab720df4846afbf2fb20b2226e7 P 2e670717494940d193cc73c7ec81a78b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a10411e514d42a6b1e9fa6d6347eb83, 2e670717494940d193cc73c7ec81a78b; no voters:
I20260924 04:59:03.842928 2719 raft_consensus.cc:2781] T a800aab720df4846afbf2fb20b2226e7 P 2e670717494940d193cc73c7ec81a78b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:59:03.843061 2719 raft_consensus.cc:686] T a800aab720df4846afbf2fb20b2226e7 P 2e670717494940d193cc73c7ec81a78b [term 1 LEADER]: Becoming Leader. State: Replica: 2e670717494940d193cc73c7ec81a78b, State: Running, Role: LEADER
I20260924 04:59:03.843106 2719 consensus_queue.cc:237] T a800aab720df4846afbf2fb20b2226e7 P 2e670717494940d193cc73c7ec81a78b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a10411e514d42a6b1e9fa6d6347eb83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } } peers { permanent_uuid: "2e670717494940d193cc73c7ec81a78b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36443 } } peers { permanent_uuid: "ff5ba82680c5444aa7f89f73e77c4504" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36967 } }
I20260924 04:59:03.843660 2119 catalog_manager.cc:5495] T a800aab720df4846afbf2fb20b2226e7 P 2e670717494940d193cc73c7ec81a78b reported cstate change: term changed from 0 to 1, leader changed from <none> to 2e670717494940d193cc73c7ec81a78b (127.0.98.2). New cstate: current_term: 1 leader_uuid: "2e670717494940d193cc73c7ec81a78b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a10411e514d42a6b1e9fa6d6347eb83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e670717494940d193cc73c7ec81a78b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36443 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff5ba82680c5444aa7f89f73e77c4504" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36967 } health_report { overall_health: UNKNOWN } } }
I20260924 04:59:03.857771 2723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:03.855801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43362 (local address 127.0.98.2:36443)
0924 04:59:03.855959 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:03.855960 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:03.855967 (+ 7us) server_negotiation.cc:407] Connection header received
0924 04:59:03.855985 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:03.855985 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:03.855994 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:03.856046 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:59:03.856152 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:03.856612 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:03.857279 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:03.857473 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:03.857513 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:03.857599 (+ 86us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:59:03.857622 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:59:03.857651 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:03.857681 (+ 30us) server_negotiation.cc:299] Negotiation successful
0924 04:59:03.857695 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":78,"threads_started":1}
I20260924 04:59:03.859756 2278 raft_consensus.cc:1261] T a800aab720df4846afbf2fb20b2226e7 P 2a10411e514d42a6b1e9fa6d6347eb83 [term 1 FOLLOWER]: Refusing update from remote peer 2e670717494940d193cc73c7ec81a78b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:59:03.859786 2558 raft_consensus.cc:1261] T a800aab720df4846afbf2fb20b2226e7 P ff5ba82680c5444aa7f89f73e77c4504 [term 1 FOLLOWER]: Refusing update from remote peer 2e670717494940d193cc73c7ec81a78b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:59:03.859941 2720 consensus_queue.cc:1035] T a800aab720df4846afbf2fb20b2226e7 P 2e670717494940d193cc73c7ec81a78b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a10411e514d42a6b1e9fa6d6347eb83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:59:03.860059 2719 consensus_queue.cc:1035] T a800aab720df4846afbf2fb20b2226e7 P 2e670717494940d193cc73c7ec81a78b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff5ba82680c5444aa7f89f73e77c4504" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:59:03.869230 2119 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59648:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:59:03.869323 2119 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59648:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:59:03.869875 2119 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cb927924e0fe40f185c2f20be51a64f0: Sending DeleteTablet for 3 replicas of tablet a800aab720df4846afbf2fb20b2226e7
I20260924 04:59:03.870079 2258 tablet_service.cc:1514] Processing DeleteTablet for tablet a800aab720df4846afbf2fb20b2226e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:59:03 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50284
I20260924 04:59:03.870143 2537 tablet_service.cc:1514] Processing DeleteTablet for tablet a800aab720df4846afbf2fb20b2226e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:59:03 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44996
I20260924 04:59:03.870143 2398 tablet_service.cc:1514] Processing DeleteTablet for tablet a800aab720df4846afbf2fb20b2226e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:59:03 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43346
I20260924 04:59:03.870170 2701 tablet_replica.cc:330] T a800aab720df4846afbf2fb20b2226e7 P 2a10411e514d42a6b1e9fa6d6347eb83: stopping tablet replica
I20260924 04:59:03.870214 2701 raft_consensus.cc:2229] T a800aab720df4846afbf2fb20b2226e7 P 2a10411e514d42a6b1e9fa6d6347eb83 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:59:03.870225 2703 tablet_replica.cc:330] T a800aab720df4846afbf2fb20b2226e7 P 2e670717494940d193cc73c7ec81a78b: stopping tablet replica
I20260924 04:59:03.870250 2701 raft_consensus.cc:2258] T a800aab720df4846afbf2fb20b2226e7 P 2a10411e514d42a6b1e9fa6d6347eb83 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:59:03.870273 2703 raft_consensus.cc:2229] T a800aab720df4846afbf2fb20b2226e7 P 2e670717494940d193cc73c7ec81a78b [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:59:03.870321 2703 raft_consensus.cc:2258] T a800aab720df4846afbf2fb20b2226e7 P 2e670717494940d193cc73c7ec81a78b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:59:03.870332 2702 tablet_replica.cc:330] T a800aab720df4846afbf2fb20b2226e7 P ff5ba82680c5444aa7f89f73e77c4504: stopping tablet replica
I20260924 04:59:03.870313 2119 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59648:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 04:59:03.870388 2702 raft_consensus.cc:2229] T a800aab720df4846afbf2fb20b2226e7 P ff5ba82680c5444aa7f89f73e77c4504 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:59:03.870416 2702 raft_consensus.cc:2258] T a800aab720df4846afbf2fb20b2226e7 P ff5ba82680c5444aa7f89f73e77c4504 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260924 04:59:03.870434 2119 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:59:03.870604 2703 ts_tablet_manager.cc:1905] T a800aab720df4846afbf2fb20b2226e7 P 2e670717494940d193cc73c7ec81a78b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:59:03.870832 2702 ts_tablet_manager.cc:1905] T a800aab720df4846afbf2fb20b2226e7 P ff5ba82680c5444aa7f89f73e77c4504: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:59:03.871505 2258 tablet_service.cc:1467] Processing CreateTablet for tablet 9e8450f6b13a41c5ad9adc392f35e455 (DEFAULT_TABLE table=test-table [id=7f2f007028a24f45988012dff98cd94b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:59:03.871606 2258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e8450f6b13a41c5ad9adc392f35e455. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:59:03.872259 2537 tablet_service.cc:1467] Processing CreateTablet for tablet 9e8450f6b13a41c5ad9adc392f35e455 (DEFAULT_TABLE table=test-table [id=7f2f007028a24f45988012dff98cd94b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:59:03.872283 2398 tablet_service.cc:1467] Processing CreateTablet for tablet 9e8450f6b13a41c5ad9adc392f35e455 (DEFAULT_TABLE table=test-table [id=7f2f007028a24f45988012dff98cd94b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:59:03.872411 2703 ts_tablet_manager.cc:1918] T a800aab720df4846afbf2fb20b2226e7 P 2e670717494940d193cc73c7ec81a78b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 04:59:03.872466 2703 log.cc:1196] T a800aab720df4846afbf2fb20b2226e7 P 2e670717494940d193cc73c7ec81a78b: Deleting WAL directory at /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225929848825-392-0/minicluster-data/ts-1/wal/wals/a800aab720df4846afbf2fb20b2226e7
I20260924 04:59:03.872660 2703 ts_tablet_manager.cc:1939] T a800aab720df4846afbf2fb20b2226e7 P 2e670717494940d193cc73c7ec81a78b: Deleting consensus metadata
I20260924 04:59:03.872797 2398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e8450f6b13a41c5ad9adc392f35e455. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:59:03.872864 2714 tablet_bootstrap.cc:492] T 9e8450f6b13a41c5ad9adc392f35e455 P 2a10411e514d42a6b1e9fa6d6347eb83: Bootstrap starting.
I20260924 04:59:03.873016 2702 ts_tablet_manager.cc:1918] T a800aab720df4846afbf2fb20b2226e7 P ff5ba82680c5444aa7f89f73e77c4504: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 04:59:03.873059 2702 log.cc:1196] T a800aab720df4846afbf2fb20b2226e7 P ff5ba82680c5444aa7f89f73e77c4504: Deleting WAL directory at /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225929848825-392-0/minicluster-data/ts-2/wal/wals/a800aab720df4846afbf2fb20b2226e7
I20260924 04:59:03.873229 2702 ts_tablet_manager.cc:1939] T a800aab720df4846afbf2fb20b2226e7 P ff5ba82680c5444aa7f89f73e77c4504: Deleting consensus metadata
I20260924 04:59:03.873342 2537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e8450f6b13a41c5ad9adc392f35e455. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:59:03.873538 2714 tablet_bootstrap.cc:654] T 9e8450f6b13a41c5ad9adc392f35e455 P 2a10411e514d42a6b1e9fa6d6347eb83: Neither blocks nor log segments found. Creating new log.
I20260924 04:59:03.873883 2713 tablet_bootstrap.cc:492] T 9e8450f6b13a41c5ad9adc392f35e455 P 2e670717494940d193cc73c7ec81a78b: Bootstrap starting.
I20260924 04:59:03.874092 2714 tablet_bootstrap.cc:492] T 9e8450f6b13a41c5ad9adc392f35e455 P 2a10411e514d42a6b1e9fa6d6347eb83: No bootstrap required, opened a new log
I20260924 04:59:03.874132 2714 ts_tablet_manager.cc:1397] T 9e8450f6b13a41c5ad9adc392f35e455 P 2a10411e514d42a6b1e9fa6d6347eb83: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260924 04:59:03.874176 2713 tablet_bootstrap.cc:654] T 9e8450f6b13a41c5ad9adc392f35e455 P 2e670717494940d193cc73c7ec81a78b: Neither blocks nor log segments found. Creating new log.
I20260924 04:59:03.874234 2714 raft_consensus.cc:348] T 9e8450f6b13a41c5ad9adc392f35e455 P 2a10411e514d42a6b1e9fa6d6347eb83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a10411e514d42a6b1e9fa6d6347eb83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } } peers { permanent_uuid: "2e670717494940d193cc73c7ec81a78b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36443 } } peers { permanent_uuid: "ff5ba82680c5444aa7f89f73e77c4504" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36967 } }
I20260924 04:59:03.874277 2714 raft_consensus.cc:374] T 9e8450f6b13a41c5ad9adc392f35e455 P 2a10411e514d42a6b1e9fa6d6347eb83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:59:03.874289 2714 raft_consensus.cc:729] T 9e8450f6b13a41c5ad9adc392f35e455 P 2a10411e514d42a6b1e9fa6d6347eb83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a10411e514d42a6b1e9fa6d6347eb83, State: Initialized, Role: FOLLOWER
I20260924 04:59:03.874315 2714 consensus_queue.cc:260] T 9e8450f6b13a41c5ad9adc392f35e455 P 2a10411e514d42a6b1e9fa6d6347eb83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a10411e514d42a6b1e9fa6d6347eb83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } } peers { permanent_uuid: "2e670717494940d193cc73c7ec81a78b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36443 } } peers { permanent_uuid: "ff5ba82680c5444aa7f89f73e77c4504" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36967 } }
I20260924 04:59:03.874367 2714 ts_tablet_manager.cc:1428] T 9e8450f6b13a41c5ad9adc392f35e455 P 2a10411e514d42a6b1e9fa6d6347eb83: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:03.874498 2106 catalog_manager.cc:4841] TS 2e670717494940d193cc73c7ec81a78b (127.0.98.2:36443): tablet a800aab720df4846afbf2fb20b2226e7 (table test-table [id=caa1b493daa34f8d95af9740c368bee5]) successfully deleted
I20260924 04:59:03.874639 2712 tablet_bootstrap.cc:492] T 9e8450f6b13a41c5ad9adc392f35e455 P ff5ba82680c5444aa7f89f73e77c4504: Bootstrap starting.
I20260924 04:59:03.874842 2107 catalog_manager.cc:4841] TS ff5ba82680c5444aa7f89f73e77c4504 (127.0.98.3:36967): tablet a800aab720df4846afbf2fb20b2226e7 (table test-table [id=caa1b493daa34f8d95af9740c368bee5]) successfully deleted
I20260924 04:59:03.874893 2712 tablet_bootstrap.cc:654] T 9e8450f6b13a41c5ad9adc392f35e455 P ff5ba82680c5444aa7f89f73e77c4504: Neither blocks nor log segments found. Creating new log.
I20260924 04:59:03.875458 2712 tablet_bootstrap.cc:492] T 9e8450f6b13a41c5ad9adc392f35e455 P ff5ba82680c5444aa7f89f73e77c4504: No bootstrap required, opened a new log
I20260924 04:59:03.875495 2713 tablet_bootstrap.cc:492] T 9e8450f6b13a41c5ad9adc392f35e455 P 2e670717494940d193cc73c7ec81a78b: No bootstrap required, opened a new log
I20260924 04:59:03.875521 2712 ts_tablet_manager.cc:1397] T 9e8450f6b13a41c5ad9adc392f35e455 P ff5ba82680c5444aa7f89f73e77c4504: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260924 04:59:03.875550 2713 ts_tablet_manager.cc:1397] T 9e8450f6b13a41c5ad9adc392f35e455 P 2e670717494940d193cc73c7ec81a78b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 04:59:03.875617 2712 raft_consensus.cc:348] T 9e8450f6b13a41c5ad9adc392f35e455 P ff5ba82680c5444aa7f89f73e77c4504 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a10411e514d42a6b1e9fa6d6347eb83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } } peers { permanent_uuid: "2e670717494940d193cc73c7ec81a78b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36443 } } peers { permanent_uuid: "ff5ba82680c5444aa7f89f73e77c4504" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36967 } }
I20260924 04:59:03.875635 2713 raft_consensus.cc:348] T 9e8450f6b13a41c5ad9adc392f35e455 P 2e670717494940d193cc73c7ec81a78b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a10411e514d42a6b1e9fa6d6347eb83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } } peers { permanent_uuid: "2e670717494940d193cc73c7ec81a78b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36443 } } peers { permanent_uuid: "ff5ba82680c5444aa7f89f73e77c4504" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36967 } }
I20260924 04:59:03.875684 2713 raft_consensus.cc:374] T 9e8450f6b13a41c5ad9adc392f35e455 P 2e670717494940d193cc73c7ec81a78b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:59:03.875682 2712 raft_consensus.cc:374] T 9e8450f6b13a41c5ad9adc392f35e455 P ff5ba82680c5444aa7f89f73e77c4504 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:59:03.875702 2713 raft_consensus.cc:729] T 9e8450f6b13a41c5ad9adc392f35e455 P 2e670717494940d193cc73c7ec81a78b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e670717494940d193cc73c7ec81a78b, State: Initialized, Role: FOLLOWER
I20260924 04:59:03.875702 2712 raft_consensus.cc:729] T 9e8450f6b13a41c5ad9adc392f35e455 P ff5ba82680c5444aa7f89f73e77c4504 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff5ba82680c5444aa7f89f73e77c4504, State: Initialized, Role: FOLLOWER
I20260924 04:59:03.875733 2713 consensus_queue.cc:260] T 9e8450f6b13a41c5ad9adc392f35e455 P 2e670717494940d193cc73c7ec81a78b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a10411e514d42a6b1e9fa6d6347eb83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } } peers { permanent_uuid: "2e670717494940d193cc73c7ec81a78b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36443 } } peers { permanent_uuid: "ff5ba82680c5444aa7f89f73e77c4504" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36967 } }
I20260924 04:59:03.875737 2712 consensus_queue.cc:260] T 9e8450f6b13a41c5ad9adc392f35e455 P ff5ba82680c5444aa7f89f73e77c4504 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a10411e514d42a6b1e9fa6d6347eb83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } } peers { permanent_uuid: "2e670717494940d193cc73c7ec81a78b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36443 } } peers { permanent_uuid: "ff5ba82680c5444aa7f89f73e77c4504" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36967 } }
I20260924 04:59:03.875814 2713 ts_tablet_manager.cc:1428] T 9e8450f6b13a41c5ad9adc392f35e455 P 2e670717494940d193cc73c7ec81a78b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:03.875814 2712 ts_tablet_manager.cc:1428] T 9e8450f6b13a41c5ad9adc392f35e455 P ff5ba82680c5444aa7f89f73e77c4504: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:03.876080 2701 ts_tablet_manager.cc:1905] T a800aab720df4846afbf2fb20b2226e7 P 2a10411e514d42a6b1e9fa6d6347eb83: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:59:03.877436 2701 ts_tablet_manager.cc:1918] T a800aab720df4846afbf2fb20b2226e7 P 2a10411e514d42a6b1e9fa6d6347eb83: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 04:59:03.877490 2701 log.cc:1196] T a800aab720df4846afbf2fb20b2226e7 P 2a10411e514d42a6b1e9fa6d6347eb83: Deleting WAL directory at /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225929848825-392-0/minicluster-data/ts-0/wal/wals/a800aab720df4846afbf2fb20b2226e7
I20260924 04:59:03.877651 2701 ts_tablet_manager.cc:1939] T a800aab720df4846afbf2fb20b2226e7 P 2a10411e514d42a6b1e9fa6d6347eb83: Deleting consensus metadata
I20260924 04:59:03.877939 2105 catalog_manager.cc:4841] TS 2a10411e514d42a6b1e9fa6d6347eb83 (127.0.98.1:38697): tablet a800aab720df4846afbf2fb20b2226e7 (table test-table [id=caa1b493daa34f8d95af9740c368bee5]) successfully deleted
I20260924 04:59:04.075623 2689 raft_consensus.cc:482] T 9e8450f6b13a41c5ad9adc392f35e455 P ff5ba82680c5444aa7f89f73e77c4504 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:59:04.075726 2689 raft_consensus.cc:504] T 9e8450f6b13a41c5ad9adc392f35e455 P ff5ba82680c5444aa7f89f73e77c4504 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a10411e514d42a6b1e9fa6d6347eb83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } } peers { permanent_uuid: "2e670717494940d193cc73c7ec81a78b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36443 } } peers { permanent_uuid: "ff5ba82680c5444aa7f89f73e77c4504" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36967 } }
I20260924 04:59:04.075907 2689 leader_election.cc:288] T 9e8450f6b13a41c5ad9adc392f35e455 P ff5ba82680c5444aa7f89f73e77c4504 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a10411e514d42a6b1e9fa6d6347eb83 (127.0.98.1:38697), 2e670717494940d193cc73c7ec81a78b (127.0.98.2:36443)
I20260924 04:59:04.076171 2277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e8450f6b13a41c5ad9adc392f35e455" candidate_uuid: "ff5ba82680c5444aa7f89f73e77c4504" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a10411e514d42a6b1e9fa6d6347eb83" is_pre_election: true
I20260924 04:59:04.076154 2418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e8450f6b13a41c5ad9adc392f35e455" candidate_uuid: "ff5ba82680c5444aa7f89f73e77c4504" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e670717494940d193cc73c7ec81a78b" is_pre_election: true
I20260924 04:59:04.076267 2418 raft_consensus.cc:2454] T 9e8450f6b13a41c5ad9adc392f35e455 P 2e670717494940d193cc73c7ec81a78b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff5ba82680c5444aa7f89f73e77c4504 in term 0.
I20260924 04:59:04.076267 2277 raft_consensus.cc:2454] T 9e8450f6b13a41c5ad9adc392f35e455 P 2a10411e514d42a6b1e9fa6d6347eb83 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff5ba82680c5444aa7f89f73e77c4504 in term 0.
I20260924 04:59:04.076432 2489 leader_election.cc:302] T 9e8450f6b13a41c5ad9adc392f35e455 P ff5ba82680c5444aa7f89f73e77c4504 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a10411e514d42a6b1e9fa6d6347eb83, ff5ba82680c5444aa7f89f73e77c4504; no voters:
I20260924 04:59:04.076545 2689 raft_consensus.cc:2781] T 9e8450f6b13a41c5ad9adc392f35e455 P ff5ba82680c5444aa7f89f73e77c4504 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:59:04.076584 2689 raft_consensus.cc:482] T 9e8450f6b13a41c5ad9adc392f35e455 P ff5ba82680c5444aa7f89f73e77c4504 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:59:04.076598 2689 raft_consensus.cc:3037] T 9e8450f6b13a41c5ad9adc392f35e455 P ff5ba82680c5444aa7f89f73e77c4504 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:59:04.077276 2689 raft_consensus.cc:504] T 9e8450f6b13a41c5ad9adc392f35e455 P ff5ba82680c5444aa7f89f73e77c4504 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a10411e514d42a6b1e9fa6d6347eb83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } } peers { permanent_uuid: "2e670717494940d193cc73c7ec81a78b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36443 } } peers { permanent_uuid: "ff5ba82680c5444aa7f89f73e77c4504" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36967 } }
I20260924 04:59:04.077402 2689 leader_election.cc:288] T 9e8450f6b13a41c5ad9adc392f35e455 P ff5ba82680c5444aa7f89f73e77c4504 [CANDIDATE]: Term 1 election: Requested vote from peers 2a10411e514d42a6b1e9fa6d6347eb83 (127.0.98.1:38697), 2e670717494940d193cc73c7ec81a78b (127.0.98.2:36443)
I20260924 04:59:04.077549 2418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e8450f6b13a41c5ad9adc392f35e455" candidate_uuid: "ff5ba82680c5444aa7f89f73e77c4504" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e670717494940d193cc73c7ec81a78b"
I20260924 04:59:04.077551 2277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e8450f6b13a41c5ad9adc392f35e455" candidate_uuid: "ff5ba82680c5444aa7f89f73e77c4504" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a10411e514d42a6b1e9fa6d6347eb83"
I20260924 04:59:04.077622 2418 raft_consensus.cc:3037] T 9e8450f6b13a41c5ad9adc392f35e455 P 2e670717494940d193cc73c7ec81a78b [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:59:04.077629 2277 raft_consensus.cc:3037] T 9e8450f6b13a41c5ad9adc392f35e455 P 2a10411e514d42a6b1e9fa6d6347eb83 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:59:04.078178 2418 raft_consensus.cc:2454] T 9e8450f6b13a41c5ad9adc392f35e455 P 2e670717494940d193cc73c7ec81a78b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff5ba82680c5444aa7f89f73e77c4504 in term 1.
I20260924 04:59:04.078264 2277 raft_consensus.cc:2454] T 9e8450f6b13a41c5ad9adc392f35e455 P 2a10411e514d42a6b1e9fa6d6347eb83 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff5ba82680c5444aa7f89f73e77c4504 in term 1.
I20260924 04:59:04.078356 2490 leader_election.cc:302] T 9e8450f6b13a41c5ad9adc392f35e455 P ff5ba82680c5444aa7f89f73e77c4504 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e670717494940d193cc73c7ec81a78b, ff5ba82680c5444aa7f89f73e77c4504; no voters:
I20260924 04:59:04.078462 2689 raft_consensus.cc:2781] T 9e8450f6b13a41c5ad9adc392f35e455 P ff5ba82680c5444aa7f89f73e77c4504 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:59:04.078523 2689 raft_consensus.cc:686] T 9e8450f6b13a41c5ad9adc392f35e455 P ff5ba82680c5444aa7f89f73e77c4504 [term 1 LEADER]: Becoming Leader. State: Replica: ff5ba82680c5444aa7f89f73e77c4504, State: Running, Role: LEADER
I20260924 04:59:04.078572 2689 consensus_queue.cc:237] T 9e8450f6b13a41c5ad9adc392f35e455 P ff5ba82680c5444aa7f89f73e77c4504 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a10411e514d42a6b1e9fa6d6347eb83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } } peers { permanent_uuid: "2e670717494940d193cc73c7ec81a78b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36443 } } peers { permanent_uuid: "ff5ba82680c5444aa7f89f73e77c4504" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36967 } }
I20260924 04:59:04.079107 2121 catalog_manager.cc:5495] T 9e8450f6b13a41c5ad9adc392f35e455 P ff5ba82680c5444aa7f89f73e77c4504 reported cstate change: term changed from 0 to 1, leader changed from <none> to ff5ba82680c5444aa7f89f73e77c4504 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "ff5ba82680c5444aa7f89f73e77c4504" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a10411e514d42a6b1e9fa6d6347eb83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e670717494940d193cc73c7ec81a78b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff5ba82680c5444aa7f89f73e77c4504" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36967 } health_report { overall_health: HEALTHY } } }
I20260924 04:59:04.102734 2732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:04.100536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45022 (local address 127.0.98.3:36967)
0924 04:59:04.100669 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:04.100671 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:04.100676 (+ 5us) server_negotiation.cc:407] Connection header received
0924 04:59:04.100692 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:04.100693 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:04.100702 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:04.100752 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:59:04.100897 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:04.101389 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:04.102125 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:04.102332 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:04.102386 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:04.102513 (+ 127us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:59:04.102538 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:59:04.102574 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:04.102624 (+ 50us) server_negotiation.cc:299] Negotiation successful
0924 04:59:04.102647 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":82,"threads_started":1}
I20260924 04:59:04.104768 2277 raft_consensus.cc:1261] T 9e8450f6b13a41c5ad9adc392f35e455 P 2a10411e514d42a6b1e9fa6d6347eb83 [term 1 FOLLOWER]: Refusing update from remote peer ff5ba82680c5444aa7f89f73e77c4504: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:59:04.104802 2418 raft_consensus.cc:1261] T 9e8450f6b13a41c5ad9adc392f35e455 P 2e670717494940d193cc73c7ec81a78b [term 1 FOLLOWER]: Refusing update from remote peer ff5ba82680c5444aa7f89f73e77c4504: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:59:04.104987 2680 consensus_queue.cc:1035] T 9e8450f6b13a41c5ad9adc392f35e455 P ff5ba82680c5444aa7f89f73e77c4504 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a10411e514d42a6b1e9fa6d6347eb83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:59:04.105103 2684 consensus_queue.cc:1035] T 9e8450f6b13a41c5ad9adc392f35e455 P ff5ba82680c5444aa7f89f73e77c4504 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e670717494940d193cc73c7ec81a78b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 04:59:04.109104 2682 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 04:59:05.625324 2120 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59648:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:59:05.625471 2120 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59648:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:59:05.626343 2120 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cb927924e0fe40f185c2f20be51a64f0: Sending DeleteTablet for 3 replicas of tablet 9e8450f6b13a41c5ad9adc392f35e455
I20260924 04:59:05.626593 2258 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e8450f6b13a41c5ad9adc392f35e455 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:59:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50284
I20260924 04:59:05.626621 2537 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e8450f6b13a41c5ad9adc392f35e455 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:59:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44996
I20260924 04:59:05.626626 2398 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e8450f6b13a41c5ad9adc392f35e455 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:59:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43346
I20260924 04:59:05.626852 2758 tablet_replica.cc:330] T 9e8450f6b13a41c5ad9adc392f35e455 P 2e670717494940d193cc73c7ec81a78b: stopping tablet replica
I20260924 04:59:05.626921 2758 raft_consensus.cc:2229] T 9e8450f6b13a41c5ad9adc392f35e455 P 2e670717494940d193cc73c7ec81a78b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:59:05.626955 2758 raft_consensus.cc:2258] T 9e8450f6b13a41c5ad9adc392f35e455 P 2e670717494940d193cc73c7ec81a78b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:59:05.626948 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 2193
I20260924 04:59:05.627556 2758 ts_tablet_manager.cc:1905] T 9e8450f6b13a41c5ad9adc392f35e455 P 2e670717494940d193cc73c7ec81a78b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:59:05.629179 2758 ts_tablet_manager.cc:1918] T 9e8450f6b13a41c5ad9adc392f35e455 P 2e670717494940d193cc73c7ec81a78b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 04:59:05.629235 2758 log.cc:1196] T 9e8450f6b13a41c5ad9adc392f35e455 P 2e670717494940d193cc73c7ec81a78b: Deleting WAL directory at /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225929848825-392-0/minicluster-data/ts-1/wal/wals/9e8450f6b13a41c5ad9adc392f35e455
I20260924 04:59:05.629403 2758 ts_tablet_manager.cc:1939] T 9e8450f6b13a41c5ad9adc392f35e455 P 2e670717494940d193cc73c7ec81a78b: Deleting consensus metadata
I20260924 04:59:05.629699 2106 catalog_manager.cc:4841] TS 2e670717494940d193cc73c7ec81a78b (127.0.98.2:36443): tablet 9e8450f6b13a41c5ad9adc392f35e455 (table test-table [id=7f2f007028a24f45988012dff98cd94b]) successfully deleted
W20260924 04:59:05.631945 2105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38697 (error 108)
I20260924 04:59:05.632110 2756 tablet_replica.cc:330] T 9e8450f6b13a41c5ad9adc392f35e455 P ff5ba82680c5444aa7f89f73e77c4504: stopping tablet replica
I20260924 04:59:05.632180 2756 raft_consensus.cc:2229] T 9e8450f6b13a41c5ad9adc392f35e455 P ff5ba82680c5444aa7f89f73e77c4504 [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:59:05.632252 2756 raft_consensus.cc:2258] T 9e8450f6b13a41c5ad9adc392f35e455 P ff5ba82680c5444aa7f89f73e77c4504 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:59:05.632380 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 2333
I20260924 04:59:05.632468 2756 ts_tablet_manager.cc:1905] T 9e8450f6b13a41c5ad9adc392f35e455 P ff5ba82680c5444aa7f89f73e77c4504: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:59:05.632962 2760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:05.632356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38697 (local address 127.0.0.1:50298)
0924 04:59:05.632876 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:05.632925 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":61,"threads_started":1}
W20260924 04:59:05.633061 2105 catalog_manager.cc:4584] TS 2a10411e514d42a6b1e9fa6d6347eb83 (127.0.98.1:38697): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9e8450f6b13a41c5ad9adc392f35e455: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38697: connect: Connection refused (error 111)
I20260924 04:59:05.634286 2756 ts_tablet_manager.cc:1918] T 9e8450f6b13a41c5ad9adc392f35e455 P ff5ba82680c5444aa7f89f73e77c4504: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 04:59:05.634343 2756 log.cc:1196] T 9e8450f6b13a41c5ad9adc392f35e455 P ff5ba82680c5444aa7f89f73e77c4504: Deleting WAL directory at /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790225929848825-392-0/minicluster-data/ts-2/wal/wals/9e8450f6b13a41c5ad9adc392f35e455
I20260924 04:59:05.634538 2756 ts_tablet_manager.cc:1939] T 9e8450f6b13a41c5ad9adc392f35e455 P ff5ba82680c5444aa7f89f73e77c4504: Deleting consensus metadata
I20260924 04:59:05.634856 2107 catalog_manager.cc:4841] TS ff5ba82680c5444aa7f89f73e77c4504 (127.0.98.3:36967): tablet 9e8450f6b13a41c5ad9adc392f35e455 (table test-table [id=7f2f007028a24f45988012dff98cd94b]) successfully deleted
I20260924 04:59:05.637971 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 2473
I20260924 04:59:05.643090 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 2089
2026-09-24T04:59:05Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5762 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225929848825-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:59:05 dist-test-slave-70st krb5kdc[2763](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:59:05 dist-test-slave-70st krb5kdc[2763](info): set up 2 sockets
Sep 24 04:59:05 dist-test-slave-70st krb5kdc[2763](info): commencing operation
krb5kdc: starting...
W20260924 04:59:06.593142 392 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225929848825-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225929848825-392-0/krb5kdc/test-user.keytab.
Sep 24 04:59:06 dist-test-slave-70st krb5kdc[2763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225946, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:59:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:59:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:59:06.704414 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225929848825-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225929848825-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225929848825-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40261
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40261
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:59:06.782676 2776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:59:06.782882 2776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:59:06.782912 2776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:59:06.784044 2776 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:59:06.784088 2776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:59:06.784106 2776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:59:06.784121 2776 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:59:06.784137 2776 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:59:06.785672 2776 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225929848825-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225929848825-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40261
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40261
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2776
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225929848825-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 04:59:06.785912 2776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:59:06.786118 2776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:59:06.790957 2781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:59:06.790946 2782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:59:06.790983 2776 server_base.cc:1034] running on GCE node
W20260924 04:59:06.790948 2784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:59:06.791464 2776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:59:06.791754 2776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:59:06.792898 2776 hybrid_clock.cc:648] HybridClock initialized: now 1790225946792884 us; error 27 us; skew 500 ppm
Sep 24 04:59:06 dist-test-slave-70st krb5kdc[2763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225946, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:59:06.795264 2776 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:59:06.795706 2776 webserver.cc:460] Webserver started at http://127.0.98.62:43541/ using document root <none> and password file <none>
I20260924 04:59:06.795914 2776 fs_manager.cc:359] Metadata directory not provided
I20260924 04:59:06.795960 2776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:59:06.796077 2776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:59:06.797075 2776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225929848825-392-0/minicluster-data/master-0/data/instance:
uuid: "2ec94535e52c40c389d4666c2d0b57a9"
format_stamp: "Formatted at 2026-09-24 04:59:06 on dist-test-slave-70st"
server_key: "c8b0b8e443657a4e904c4146b93a02bc"
server_key_iv: "cb77139a1effc39c2282caea57f01f40"
server_key_version: "encryptionkey@0"
I20260924 04:59:06.797394 2776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225929848825-392-0/minicluster-data/master-0/wal/instance:
uuid: "2ec94535e52c40c389d4666c2d0b57a9"
format_stamp: "Formatted at 2026-09-24 04:59:06 on dist-test-slave-70st"
server_key: "c8b0b8e443657a4e904c4146b93a02bc"
server_key_iv: "cb77139a1effc39c2282caea57f01f40"
server_key_version: "encryptionkey@0"
I20260924 04:59:06.798597 2776 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 04:59:06.799304 2791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:06.799507 2776 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 04:59:06.799597 2776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225929848825-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225929848825-392-0/minicluster-data/master-0/wal
uuid: "2ec94535e52c40c389d4666c2d0b57a9"
format_stamp: "Formatted at 2026-09-24 04:59:06 on dist-test-slave-70st"
server_key: "c8b0b8e443657a4e904c4146b93a02bc"
server_key_iv: "cb77139a1effc39c2282caea57f01f40"
server_key_version: "encryptionkey@0"
I20260924 04:59:06.799657 2776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225929848825-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225929848825-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225929848825-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:59:06.817063 2776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:59:06.819304 2776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:59:06.819456 2776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:59:06.823027 2776 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40261
I20260924 04:59:06.823067 2843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40261 every 8 connection(s)
I20260924 04:59:06.823529 2776 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:59:06.824179 2844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:59:06.826496 2844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ec94535e52c40c389d4666c2d0b57a9: Bootstrap starting.
I20260924 04:59:06.827049 2844 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2ec94535e52c40c389d4666c2d0b57a9: Neither blocks nor log segments found. Creating new log.
I20260924 04:59:06.827297 2844 log.cc:824] T 00000000000000000000000000000000 P 2ec94535e52c40c389d4666c2d0b57a9: Log is configured to *not* fsync() on all Append() calls
I20260924 04:59:06.827888 2844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ec94535e52c40c389d4666c2d0b57a9: No bootstrap required, opened a new log
I20260924 04:59:06.828681 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 2776
I20260924 04:59:06.828747 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225929848825-392-0/minicluster-data/master-0/wal/instance
I20260924 04:59:06.828989 392 external_mini_cluster.cc:1428] Setting key e29a92ce694f5064ba666b6c93102896
I20260924 04:59:06.830279 2844 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ec94535e52c40c389d4666c2d0b57a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ec94535e52c40c389d4666c2d0b57a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40261 } }
I20260924 04:59:06.830422 2844 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2ec94535e52c40c389d4666c2d0b57a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:59:06.830454 2844 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ec94535e52c40c389d4666c2d0b57a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ec94535e52c40c389d4666c2d0b57a9, State: Initialized, Role: FOLLOWER
I20260924 04:59:06.830595 2844 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ec94535e52c40c389d4666c2d0b57a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ec94535e52c40c389d4666c2d0b57a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40261 } }
I20260924 04:59:06.830669 2844 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2ec94535e52c40c389d4666c2d0b57a9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:59:06.830700 2844 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ec94535e52c40c389d4666c2d0b57a9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:59:06.830749 2844 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ec94535e52c40c389d4666c2d0b57a9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:59:06.831257 2844 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ec94535e52c40c389d4666c2d0b57a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ec94535e52c40c389d4666c2d0b57a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40261 } }
I20260924 04:59:06.831352 2844 leader_election.cc:302] T 00000000000000000000000000000000 P 2ec94535e52c40c389d4666c2d0b57a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ec94535e52c40c389d4666c2d0b57a9; no voters:
I20260924 04:59:06.831497 2844 leader_election.cc:288] T 00000000000000000000000000000000 P 2ec94535e52c40c389d4666c2d0b57a9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:59:06.831534 2849 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2ec94535e52c40c389d4666c2d0b57a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:59:06.831633 2849 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2ec94535e52c40c389d4666c2d0b57a9 [term 1 LEADER]: Becoming Leader. State: Replica: 2ec94535e52c40c389d4666c2d0b57a9, State: Running, Role: LEADER
I20260924 04:59:06.831753 2844 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ec94535e52c40c389d4666c2d0b57a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:59:06.831772 2849 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ec94535e52c40c389d4666c2d0b57a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ec94535e52c40c389d4666c2d0b57a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40261 } }
I20260924 04:59:06.832170 2849 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ec94535e52c40c389d4666c2d0b57a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ec94535e52c40c389d4666c2d0b57a9. Latest consensus state: current_term: 1 leader_uuid: "2ec94535e52c40c389d4666c2d0b57a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ec94535e52c40c389d4666c2d0b57a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40261 } } }
I20260924 04:59:06.832218 2850 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ec94535e52c40c389d4666c2d0b57a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2ec94535e52c40c389d4666c2d0b57a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ec94535e52c40c389d4666c2d0b57a9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40261 } } }
I20260924 04:59:06.832368 2849 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ec94535e52c40c389d4666c2d0b57a9 [sys.catalog]: This master's current role is: LEADER
I20260924 04:59:06.832382 2850 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ec94535e52c40c389d4666c2d0b57a9 [sys.catalog]: This master's current role is: LEADER
I20260924 04:59:06.833205 2858 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:59:06.834034 2858 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:59:06.835605 2858 catalog_manager.cc:1347] Generated new cluster ID: 2bc79cbc89bf42559f1a11ac1ba75cba
I20260924 04:59:06.835650 2858 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 04:59:06 dist-test-slave-70st krb5kdc[2763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225946, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:59:06.839644 2848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:06.829564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34330 (local address 127.0.98.62:40261)
0924 04:59:06.829784 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:06.829792 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:06.829817 (+ 25us) server_negotiation.cc:407] Connection header received
0924 04:59:06.830265 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:06.830272 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:06.830323 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:06.830456 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:06.831180 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:06.832361 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:06.833461 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:06.833945 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:06.834048 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:06.836643 (+ 2595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:06.836647 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:06.836648 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:59:06.836658 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:06.838256 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:06.838563 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:06.838566 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:06.838566 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:06.838664 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:06.838838 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:06.838839 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:06.838839 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:06.839057 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:06.839208 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:06.839282 (+ 74us) server_negotiation.cc:299] Negotiation successful
0924 04:59:06.839501 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":96,"threads_started":1}
I20260924 04:59:06.848038 2858 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:59:06.848570 2858 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:59:06.859900 2858 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2ec94535e52c40c389d4666c2d0b57a9: Generated new TSK 0
I20260924 04:59:06.860076 2858 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225929848825-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225929848825-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:59:06.884137 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225929848825-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225929848825-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225929848825-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225929848825-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40261
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225929848825-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:59:06.962800 2870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:59:06.962994 2870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:59:06.963017 2870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:59:06.964147 2870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:59:06.964190 2870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:59:06.964219 2870 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:59:06.965713 2870 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225929848825-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225929848825-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225929848825-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225929848825-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40261
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2870
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225929848825-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 04:59:06.965931 2870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:59:06.966138 2870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:59:06.970839 2876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:59:06.970839 2878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:59:06.970903 2875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:59:06.971105 2870 server_base.cc:1034] running on GCE node
I20260924 04:59:06.971315 2870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:59:06.971525 2870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:59:06.972657 2870 hybrid_clock.cc:648] HybridClock initialized: now 1790225946972647 us; error 26 us; skew 500 ppm
Sep 24 04:59:06 dist-test-slave-70st krb5kdc[2763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225946, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:59:06.975116 2870 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:59:06.975584 2870 webserver.cc:460] Webserver started at http://127.0.98.1:34825/ using document root <none> and password file <none>
I20260924 04:59:06.975795 2870 fs_manager.cc:359] Metadata directory not provided
I20260924 04:59:06.975840 2870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:59:06.975962 2870 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:59:06.976981 2870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225929848825-392-0/minicluster-data/ts-0/data/instance:
uuid: "bae3ef2927c94599b91370dbd6e2fc51"
format_stamp: "Formatted at 2026-09-24 04:59:06 on dist-test-slave-70st"
server_key: "3f282b6c1416bc3a9f65e1f13086295d"
server_key_iv: "fce8af509e556ea78572eec4b8dc9c5c"
server_key_version: "encryptionkey@0"
I20260924 04:59:06.977298 2870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225929848825-392-0/minicluster-data/ts-0/wal/instance:
uuid: "bae3ef2927c94599b91370dbd6e2fc51"
format_stamp: "Formatted at 2026-09-24 04:59:06 on dist-test-slave-70st"
server_key: "3f282b6c1416bc3a9f65e1f13086295d"
server_key_iv: "fce8af509e556ea78572eec4b8dc9c5c"
server_key_version: "encryptionkey@0"
I20260924 04:59:06.978528 2870 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 04:59:06.979138 2885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:06.979293 2870 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 04:59:06.979372 2870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225929848825-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225929848825-392-0/minicluster-data/ts-0/wal
uuid: "bae3ef2927c94599b91370dbd6e2fc51"
format_stamp: "Formatted at 2026-09-24 04:59:06 on dist-test-slave-70st"
server_key: "3f282b6c1416bc3a9f65e1f13086295d"
server_key_iv: "fce8af509e556ea78572eec4b8dc9c5c"
server_key_version: "encryptionkey@0"
I20260924 04:59:06.979434 2870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225929848825-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225929848825-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225929848825-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:59:06.993101 2870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:59:06.995172 2870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:59:06.995301 2870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:59:06.995505 2870 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:59:06.995771 2870 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:59:06.995805 2870 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:06.995832 2870 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:59:06.995849 2870 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:07.001252 2870 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40179
I20260924 04:59:07.001307 2998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40179 every 8 connection(s)
I20260924 04:59:07.001632 2870 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225929848825-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 04:59:07 dist-test-slave-70st krb5kdc[2763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225946, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:59:07.008487 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 2870
I20260924 04:59:07.008572 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225929848825-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:59:07.008757 392 external_mini_cluster.cc:1428] Setting key 150201463e3c9610b54fcbdb1aac0377
I20260924 04:59:07.010571 2848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:07.002319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49445 (local address 127.0.98.62:40261)
0924 04:59:07.002369 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:07.002371 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:07.002590 (+ 219us) server_negotiation.cc:407] Connection header received
0924 04:59:07.003581 (+ 991us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:07.003582 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:07.003598 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:07.003647 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:07.004250 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:07.004745 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:07.005761 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:07.005982 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:07.006040 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:07.007924 (+ 1884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:07.007925 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:07.007925 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:59:07.007932 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:07.009271 (+ 1339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:07.009702 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:07.009703 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:07.009704 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:07.009761 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:07.009968 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:07.009969 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:07.009970 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:07.010079 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:07.010133 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:07.010382 (+ 249us) server_negotiation.cc:299] Negotiation successful
0924 04:59:07.010476 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 04:59:07.010710 3001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:07.002313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40261 (local address 127.0.98.1:49445)
0924 04:59:07.002478 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:07.002512 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:07.003483 (+ 971us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:07.003716 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:07.003720 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:07.003975 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:07.004158 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:07.004162 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:07.004822 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:07.004823 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:07.005684 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:07.005686 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:07.006051 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:07.006052 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:07.006169 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:07.006589 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:07.006598 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:07.007832 (+ 1234us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:59:07.009383 (+ 1551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:07.009389 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:07.009391 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:07.009634 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:07.009813 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:07.009814 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:07.009814 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:07.009913 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:07.010131 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:59:07.010137 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:59:07.010265 (+ 128us) client_negotiation.cc:769] Sending connection context
0924 04:59:07.010357 (+ 92us) client_negotiation.cc:240] Negotiation successful
0924 04:59:07.010504 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":76,"threads_started":1}
I20260924 04:59:07.011199 2999 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40261
I20260924 04:59:07.011294 2999 heartbeater.cc:464] Registering TS with master...
I20260924 04:59:07.011463 2999 heartbeater.cc:511] Master 127.0.98.62:40261 requested a full tablet report, sending...
I20260924 04:59:07.011981 2808 ts_manager.cc:194] Registered new tserver with Master: bae3ef2927c94599b91370dbd6e2fc51 (127.0.98.1:40179)
I20260924 04:59:07.012847 2808 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:49445
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225929848825-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225929848825-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:59:07.033870 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225929848825-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225929848825-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225929848825-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225929848825-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40261
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225929848825-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:59:07.112612 3004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:59:07.112840 3004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:59:07.112865 3004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:59:07.114058 3004 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:59:07.114104 3004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:59:07.114130 3004 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:59:07.115666 3004 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225929848825-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225929848825-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225929848825-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225929848825-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40261
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3004
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225929848825-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 04:59:07.115937 3004 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:59:07.116160 3004 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:59:07.120929 3009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:59:07.120926 3012 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:59:07.120932 3010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:59:07.121146 3004 server_base.cc:1034] running on GCE node
I20260924 04:59:07.121376 3004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:59:07.121635 3004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:59:07.122807 3004 hybrid_clock.cc:648] HybridClock initialized: now 1790225947122784 us; error 30 us; skew 500 ppm
Sep 24 04:59:07 dist-test-slave-70st krb5kdc[2763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225947, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:59:07.125348 3004 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:59:07.125808 3004 webserver.cc:460] Webserver started at http://127.0.98.2:36889/ using document root <none> and password file <none>
I20260924 04:59:07.126024 3004 fs_manager.cc:359] Metadata directory not provided
I20260924 04:59:07.126072 3004 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:59:07.126225 3004 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:59:07.127461 3004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225929848825-392-0/minicluster-data/ts-1/data/instance:
uuid: "724475c5e29740e6ae9f3036cff733f6"
format_stamp: "Formatted at 2026-09-24 04:59:07 on dist-test-slave-70st"
server_key: "ec390bb4e89aceba085169195e5efa32"
server_key_iv: "2e380e8b3ce11966cf5a220495c7ca96"
server_key_version: "encryptionkey@0"
I20260924 04:59:07.127794 3004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225929848825-392-0/minicluster-data/ts-1/wal/instance:
uuid: "724475c5e29740e6ae9f3036cff733f6"
format_stamp: "Formatted at 2026-09-24 04:59:07 on dist-test-slave-70st"
server_key: "ec390bb4e89aceba085169195e5efa32"
server_key_iv: "2e380e8b3ce11966cf5a220495c7ca96"
server_key_version: "encryptionkey@0"
I20260924 04:59:07.129099 3004 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 04:59:07.129863 3019 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:07.130035 3004 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:07.130126 3004 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225929848825-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225929848825-392-0/minicluster-data/ts-1/wal
uuid: "724475c5e29740e6ae9f3036cff733f6"
format_stamp: "Formatted at 2026-09-24 04:59:07 on dist-test-slave-70st"
server_key: "ec390bb4e89aceba085169195e5efa32"
server_key_iv: "2e380e8b3ce11966cf5a220495c7ca96"
server_key_version: "encryptionkey@0"
I20260924 04:59:07.130187 3004 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225929848825-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225929848825-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225929848825-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:59:07.143194 3004 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:59:07.145251 3004 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:59:07.145411 3004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:59:07.145650 3004 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:59:07.145933 3004 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:59:07.145967 3004 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:07.145993 3004 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:59:07.146016 3004 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:07.151494 3004 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35973
I20260924 04:59:07.151592 3132 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35973 every 8 connection(s)
I20260924 04:59:07.151866 3004 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225929848825-392-0/minicluster-data/ts-1/data/info.pb
Sep 24 04:59:07 dist-test-slave-70st krb5kdc[2763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225947, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:59:07.158463 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 3004
I20260924 04:59:07.158563 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225929848825-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:59:07.158735 392 external_mini_cluster.cc:1428] Setting key c613219ec2b0e490227b43337474d018
I20260924 04:59:07.160861 2848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:07.152586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53997 (local address 127.0.98.62:40261)
0924 04:59:07.152665 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:07.152668 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:07.152821 (+ 153us) server_negotiation.cc:407] Connection header received
0924 04:59:07.153817 (+ 996us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:07.153819 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:07.153829 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:07.153872 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:07.154513 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:07.155037 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:07.156034 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:07.156220 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:07.156272 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:07.158261 (+ 1989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:07.158262 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:07.158262 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:59:07.158269 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:07.159663 (+ 1394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:07.160022 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:07.160023 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:07.160023 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:07.160074 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:07.160279 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:07.160280 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:07.160280 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:07.160385 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:07.160438 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:07.160691 (+ 253us) server_negotiation.cc:299] Negotiation successful
0924 04:59:07.160771 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 04:59:07.161043 3135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:07.152556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40261 (local address 127.0.98.2:53997)
0924 04:59:07.152697 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:07.152733 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:07.153686 (+ 953us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:07.153944 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:07.153947 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:07.154194 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:07.154408 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:07.154414 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:07.155110 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:07.155110 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:07.155963 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:07.155965 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:07.156279 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:07.156279 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:07.156378 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:07.156824 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:07.156831 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:07.158185 (+ 1354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:59:07.159732 (+ 1547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:07.159739 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:07.159742 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:07.159959 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:07.160125 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:07.160125 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:07.160126 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:07.160228 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:07.160437 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:59:07.160443 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:59:07.160579 (+ 136us) client_negotiation.cc:769] Sending connection context
0924 04:59:07.160667 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 04:59:07.160824 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":67,"threads_started":1}
I20260924 04:59:07.161546 3133 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40261
I20260924 04:59:07.161638 3133 heartbeater.cc:464] Registering TS with master...
I20260924 04:59:07.161823 3133 heartbeater.cc:511] Master 127.0.98.62:40261 requested a full tablet report, sending...
I20260924 04:59:07.162186 2808 ts_manager.cc:194] Registered new tserver with Master: 724475c5e29740e6ae9f3036cff733f6 (127.0.98.2:35973)
I20260924 04:59:07.162617 2808 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53997
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225929848825-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225929848825-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:59:07.182087 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225929848825-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225929848825-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225929848825-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225929848825-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40261
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225929848825-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:59:07.261175 3138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:59:07.261385 3138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:59:07.261447 3138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:59:07.262625 3138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:59:07.262668 3138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:59:07.262703 3138 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:59:07.264209 3138 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225929848825-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225929848825-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225929848825-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225929848825-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40261
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3138
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225929848825-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 04:59:07.264447 3138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:59:07.264657 3138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:59:07.269469 3143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:59:07.269469 3146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:59:07.269546 3144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:59:07.269609 3138 server_base.cc:1034] running on GCE node
I20260924 04:59:07.269896 3138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:59:07.270128 3138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:59:07.271271 3138 hybrid_clock.cc:648] HybridClock initialized: now 1790225947271256 us; error 27 us; skew 500 ppm
Sep 24 04:59:07 dist-test-slave-70st krb5kdc[2763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225947, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:59:07.273625 3138 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:59:07.274055 3138 webserver.cc:460] Webserver started at http://127.0.98.3:45679/ using document root <none> and password file <none>
I20260924 04:59:07.274252 3138 fs_manager.cc:359] Metadata directory not provided
I20260924 04:59:07.274304 3138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:59:07.274433 3138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:59:07.275477 3138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225929848825-392-0/minicluster-data/ts-2/data/instance:
uuid: "a5f35fade8d146b287970350ba1f2288"
format_stamp: "Formatted at 2026-09-24 04:59:07 on dist-test-slave-70st"
server_key: "adf03826afe52ca1c0a19855040c02b6"
server_key_iv: "b4fae64312c6c929b2547be749bd1346"
server_key_version: "encryptionkey@0"
I20260924 04:59:07.275813 3138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225929848825-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a5f35fade8d146b287970350ba1f2288"
format_stamp: "Formatted at 2026-09-24 04:59:07 on dist-test-slave-70st"
server_key: "adf03826afe52ca1c0a19855040c02b6"
server_key_iv: "b4fae64312c6c929b2547be749bd1346"
server_key_version: "encryptionkey@0"
I20260924 04:59:07.277078 3138 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 04:59:07.277783 3153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:07.277931 3138 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:07.278010 3138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225929848825-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225929848825-392-0/minicluster-data/ts-2/wal
uuid: "a5f35fade8d146b287970350ba1f2288"
format_stamp: "Formatted at 2026-09-24 04:59:07 on dist-test-slave-70st"
server_key: "adf03826afe52ca1c0a19855040c02b6"
server_key_iv: "b4fae64312c6c929b2547be749bd1346"
server_key_version: "encryptionkey@0"
I20260924 04:59:07.278069 3138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225929848825-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225929848825-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225929848825-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:59:07.285064 3138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:59:07.286849 3138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:59:07.286979 3138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:59:07.287174 3138 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:59:07.287456 3138 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:59:07.287488 3138 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:07.287516 3138 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:59:07.287539 3138 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:07.292954 3138 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36487
I20260924 04:59:07.292992 3266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36487 every 8 connection(s)
I20260924 04:59:07.293292 3138 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225929848825-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:59:07.296519 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 3138
I20260924 04:59:07.296602 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790225929848825-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:59:07.296769 392 external_mini_cluster.cc:1428] Setting key 87da120c85cf068bea8bb27f2e26289c
Sep 24 04:59:07 dist-test-slave-70st krb5kdc[2763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225947, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:59:07.302057 2848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:07.293934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56149 (local address 127.0.98.62:40261)
0924 04:59:07.293976 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:07.293978 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:07.294173 (+ 195us) server_negotiation.cc:407] Connection header received
0924 04:59:07.295081 (+ 908us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:07.295082 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:07.295091 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:07.295136 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:07.295708 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:07.296213 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:07.297384 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:07.297641 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:07.297705 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:07.299600 (+ 1895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:07.299601 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:07.299601 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:59:07.299607 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:07.300917 (+ 1310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:07.301258 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:07.301259 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:07.301259 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:07.301306 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:07.301514 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:07.301514 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:07.301514 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:07.301615 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:07.301678 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:07.301932 (+ 254us) server_negotiation.cc:299] Negotiation successful
0924 04:59:07.301990 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 04:59:07.302266 3269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:07.293935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40261 (local address 127.0.98.3:56149)
0924 04:59:07.294057 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:07.294091 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:07.294997 (+ 906us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:07.295200 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:07.295203 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:07.295455 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:07.295649 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:07.295653 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:07.296278 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:07.296279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:07.297317 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:07.297320 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:07.297722 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:07.297723 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:07.297826 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:07.298240 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:07.298247 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:07.299515 (+ 1268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:59:07.300984 (+ 1469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:07.300990 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:07.300992 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:07.301203 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:07.301351 (+ 148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:07.301351 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:07.301351 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:07.301473 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:07.301673 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:59:07.301677 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:59:07.301795 (+ 118us) client_negotiation.cc:769] Sending connection context
0924 04:59:07.301901 (+ 106us) client_negotiation.cc:240] Negotiation successful
0924 04:59:07.302038 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":65,"threads_started":1}
I20260924 04:59:07.302718 3267 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40261
I20260924 04:59:07.302802 3267 heartbeater.cc:464] Registering TS with master...
I20260924 04:59:07.302999 3267 heartbeater.cc:511] Master 127.0.98.62:40261 requested a full tablet report, sending...
I20260924 04:59:07.303375 2808 ts_manager.cc:194] Registered new tserver with Master: a5f35fade8d146b287970350ba1f2288 (127.0.98.3:36487)
I20260924 04:59:07.303711 2808 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:56149
I20260924 04:59:07.308254 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 04:59:07.318164 2848 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:59:07.317264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34332 (local address 127.0.98.62:40261)
0924 04:59:07.317322 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:07.317324 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:07.317441 (+ 117us) server_negotiation.cc:407] Connection header received
0924 04:59:07.317554 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:07.317555 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:07.317567 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:07.317616 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:07.318094 (+ 478us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34332: BlockingRecv error: recv got EOF from 127.0.0.1:34332 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 04:59:07.319135 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 2870
I20260924 04:59:07.322824 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 3004
I20260924 04:59:07.326217 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 3138
I20260924 04:59:07.329596 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 2776
2026-09-24T04:59:07Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1685 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790225929848825-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:59:07 dist-test-slave-70st krb5kdc[3279](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:59:07 dist-test-slave-70st krb5kdc[3279](info): set up 2 sockets
Sep 24 04:59:07 dist-test-slave-70st krb5kdc[3279](info): commencing operation
krb5kdc: starting...
W20260924 04:59:08.281014 392 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.004s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790225929848825-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790225929848825-392-0/krb5kdc/test-user.keytab.
Sep 24 04:59:08 dist-test-slave-70st krb5kdc[3279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225948, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:59:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:59:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:59:08.391444 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790225929848825-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790225929848825-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790225929848825-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40663
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40663
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:59:08.469686 3292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:59:08.469882 3292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:59:08.469902 3292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:59:08.470985 3292 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:59:08.471017 3292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:59:08.471029 3292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:59:08.471040 3292 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:59:08.471055 3292 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:59:08.472446 3292 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790225929848825-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790225929848825-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40663
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40663
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790225929848825-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 04:59:08.472630 3292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:59:08.472795 3292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:59:08.477241 3298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:59:08.477403 3292 server_base.cc:1034] running on GCE node
W20260924 04:59:08.477263 3297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:59:08.477286 3300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:59:08.477762 3292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:59:08.478036 3292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:59:08.479177 3292 hybrid_clock.cc:648] HybridClock initialized: now 1790225948479159 us; error 32 us; skew 500 ppm
Sep 24 04:59:08 dist-test-slave-70st krb5kdc[3279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225948, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:59:08.481526 3292 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:59:08.481943 3292 webserver.cc:460] Webserver started at http://127.0.98.62:44213/ using document root <none> and password file <none>
I20260924 04:59:08.482139 3292 fs_manager.cc:359] Metadata directory not provided
I20260924 04:59:08.482185 3292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:59:08.482306 3292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:59:08.483320 3292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790225929848825-392-0/minicluster-data/master-0/data/instance:
uuid: "4f0c90f17ba945aa89d06a0c68645a45"
format_stamp: "Formatted at 2026-09-24 04:59:08 on dist-test-slave-70st"
server_key: "bdd358541704759ae50b7f50cc7d9f80"
server_key_iv: "a33ec034ec9fcae95805ebef68f1317c"
server_key_version: "encryptionkey@0"
I20260924 04:59:08.483647 3292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790225929848825-392-0/minicluster-data/master-0/wal/instance:
uuid: "4f0c90f17ba945aa89d06a0c68645a45"
format_stamp: "Formatted at 2026-09-24 04:59:08 on dist-test-slave-70st"
server_key: "bdd358541704759ae50b7f50cc7d9f80"
server_key_iv: "a33ec034ec9fcae95805ebef68f1317c"
server_key_version: "encryptionkey@0"
I20260924 04:59:08.484817 3292 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 04:59:08.485491 3307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:08.485641 3292 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:08.485720 3292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790225929848825-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790225929848825-392-0/minicluster-data/master-0/wal
uuid: "4f0c90f17ba945aa89d06a0c68645a45"
format_stamp: "Formatted at 2026-09-24 04:59:08 on dist-test-slave-70st"
server_key: "bdd358541704759ae50b7f50cc7d9f80"
server_key_iv: "a33ec034ec9fcae95805ebef68f1317c"
server_key_version: "encryptionkey@0"
I20260924 04:59:08.485785 3292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790225929848825-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790225929848825-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790225929848825-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:59:08.504009 3292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:59:08.506112 3292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:59:08.506247 3292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:59:08.509835 3292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40663
I20260924 04:59:08.509869 3359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40663 every 8 connection(s)
I20260924 04:59:08.510186 3292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:59:08.510985 3360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:59:08.513214 3360 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f0c90f17ba945aa89d06a0c68645a45: Bootstrap starting.
I20260924 04:59:08.513814 3360 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4f0c90f17ba945aa89d06a0c68645a45: Neither blocks nor log segments found. Creating new log.
I20260924 04:59:08.514079 3360 log.cc:824] T 00000000000000000000000000000000 P 4f0c90f17ba945aa89d06a0c68645a45: Log is configured to *not* fsync() on all Append() calls
I20260924 04:59:08.514678 3360 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f0c90f17ba945aa89d06a0c68645a45: No bootstrap required, opened a new log
I20260924 04:59:08.515671 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 3292
I20260924 04:59:08.515741 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790225929848825-392-0/minicluster-data/master-0/wal/instance
I20260924 04:59:08.515893 392 external_mini_cluster.cc:1428] Setting key 97f9727e3d2e5fb0cf21557ae657b5aa
I20260924 04:59:08.516372 3360 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4f0c90f17ba945aa89d06a0c68645a45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f0c90f17ba945aa89d06a0c68645a45" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40663 } }
I20260924 04:59:08.516518 3360 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4f0c90f17ba945aa89d06a0c68645a45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:59:08.516551 3360 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4f0c90f17ba945aa89d06a0c68645a45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f0c90f17ba945aa89d06a0c68645a45, State: Initialized, Role: FOLLOWER
I20260924 04:59:08.516677 3360 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4f0c90f17ba945aa89d06a0c68645a45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f0c90f17ba945aa89d06a0c68645a45" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40663 } }
I20260924 04:59:08.516744 3360 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4f0c90f17ba945aa89d06a0c68645a45 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:59:08.516763 3360 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4f0c90f17ba945aa89d06a0c68645a45 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:59:08.516798 3360 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4f0c90f17ba945aa89d06a0c68645a45 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:59:08.517378 3360 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4f0c90f17ba945aa89d06a0c68645a45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f0c90f17ba945aa89d06a0c68645a45" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40663 } }
I20260924 04:59:08.517508 3360 leader_election.cc:302] T 00000000000000000000000000000000 P 4f0c90f17ba945aa89d06a0c68645a45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4f0c90f17ba945aa89d06a0c68645a45; no voters:
I20260924 04:59:08.517674 3360 leader_election.cc:288] T 00000000000000000000000000000000 P 4f0c90f17ba945aa89d06a0c68645a45 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:59:08.517699 3365 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4f0c90f17ba945aa89d06a0c68645a45 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:59:08.517820 3365 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4f0c90f17ba945aa89d06a0c68645a45 [term 1 LEADER]: Becoming Leader. State: Replica: 4f0c90f17ba945aa89d06a0c68645a45, State: Running, Role: LEADER
I20260924 04:59:08.517933 3360 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4f0c90f17ba945aa89d06a0c68645a45 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:59:08.517939 3365 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4f0c90f17ba945aa89d06a0c68645a45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f0c90f17ba945aa89d06a0c68645a45" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40663 } }
I20260924 04:59:08.518325 3366 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f0c90f17ba945aa89d06a0c68645a45 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4f0c90f17ba945aa89d06a0c68645a45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f0c90f17ba945aa89d06a0c68645a45" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40663 } } }
I20260924 04:59:08.518325 3367 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f0c90f17ba945aa89d06a0c68645a45 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f0c90f17ba945aa89d06a0c68645a45. Latest consensus state: current_term: 1 leader_uuid: "4f0c90f17ba945aa89d06a0c68645a45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f0c90f17ba945aa89d06a0c68645a45" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40663 } } }
I20260924 04:59:08.518483 3366 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f0c90f17ba945aa89d06a0c68645a45 [sys.catalog]: This master's current role is: LEADER
I20260924 04:59:08.518489 3367 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f0c90f17ba945aa89d06a0c68645a45 [sys.catalog]: This master's current role is: LEADER
I20260924 04:59:08.518699 3374 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:59:08.519225 3374 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:59:08.522583 3374 catalog_manager.cc:1347] Generated new cluster ID: 2f222361363a4bd6b436ebcef4a0303d
I20260924 04:59:08.522636 3374 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 04:59:08 dist-test-slave-70st krb5kdc[3279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225948, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:59:08.526708 3364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:08.516354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38924 (local address 127.0.98.62:40663)
0924 04:59:08.516876 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:08.516883 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:08.516907 (+ 24us) server_negotiation.cc:407] Connection header received
0924 04:59:08.517007 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:08.517013 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:08.517057 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:08.517161 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:08.517958 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:08.519424 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:08.520274 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:08.520862 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:08.520973 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:08.523416 (+ 2443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:08.523421 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:08.523421 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:59:08.523432 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:08.525301 (+ 1869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:08.525653 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:08.525656 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:08.525656 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:08.525744 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:08.525919 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:08.525920 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:08.525920 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:08.526157 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:08.526296 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:08.526394 (+ 98us) server_negotiation.cc:299] Negotiation successful
0924 04:59:08.526541 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":441,"threads_started":1}
I20260924 04:59:08.527508 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:59:08.529451 3374 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:59:08.530092 3374 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:59:08.537083 3374 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4f0c90f17ba945aa89d06a0c68645a45: Generated new TSK 0
I20260924 04:59:08.537314 3374 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:59:08.545271 3364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:08.541627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38936 (local address 127.0.98.62:40663)
0924 04:59:08.541670 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:08.541672 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:08.541826 (+ 154us) server_negotiation.cc:407] Connection header received
0924 04:59:08.541899 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:08.541900 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:08.541910 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:08.541961 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:08.542365 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:08.542871 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:08.543942 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:08.544131 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:08.544177 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:08.544836 (+ 659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:08.544837 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:08.544838 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:59:08.545041 (+ 203us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:08.545059 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:59:08.545065 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:08.545108 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:08.545179 (+ 71us) server_negotiation.cc:299] Negotiation successful
0924 04:59:08.545204 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 04:59:08.547703 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 3292
2026-09-24T04:59:08Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1216 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225929848825-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:59:08 dist-test-slave-70st krb5kdc[3393](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:59:08 dist-test-slave-70st krb5kdc[3393](info): set up 2 sockets
Sep 24 04:59:08 dist-test-slave-70st krb5kdc[3393](info): commencing operation
krb5kdc: starting...
W20260924 04:59:09.505033 392 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225929848825-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225929848825-392-0/krb5kdc/test-user.keytab.
Sep 24 04:59:09 dist-test-slave-70st krb5kdc[3393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225949, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:59:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:59:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:59:09.616707 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225929848825-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225929848825-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225929848825-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43219
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43219
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:59:09.694152 3406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:59:09.694347 3406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:59:09.694368 3406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:59:09.695475 3406 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:59:09.695513 3406 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:59:09.695525 3406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:59:09.695536 3406 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:59:09.695547 3406 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:59:09.696938 3406 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225929848825-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225929848825-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43219
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3406
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225929848825-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 04:59:09.697127 3406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:59:09.697306 3406 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:59:09.701949 3414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:59:09.701975 3412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:59:09.702008 3411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:59:09.702088 3406 server_base.cc:1034] running on GCE node
I20260924 04:59:09.702380 3406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:59:09.702682 3406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:59:09.703827 3406 hybrid_clock.cc:648] HybridClock initialized: now 1790225949703813 us; error 31 us; skew 500 ppm
Sep 24 04:59:09 dist-test-slave-70st krb5kdc[3393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225949, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:59:09.718683 3406 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:59:09.719135 3406 webserver.cc:460] Webserver started at http://127.0.98.62:38137/ using document root <none> and password file <none>
I20260924 04:59:09.719362 3406 fs_manager.cc:359] Metadata directory not provided
I20260924 04:59:09.719409 3406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:59:09.719532 3406 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:59:09.720603 3406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225929848825-392-0/minicluster-data/master-0/data/instance:
uuid: "082feed59e4143bfaa3839bade529fb1"
format_stamp: "Formatted at 2026-09-24 04:59:09 on dist-test-slave-70st"
server_key: "fbea1c649036ca49d00e5dbf71dba37c"
server_key_iv: "2cccd3592588be2f6e0adc87494a5012"
server_key_version: "encryptionkey@0"
I20260924 04:59:09.720961 3406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225929848825-392-0/minicluster-data/master-0/wal/instance:
uuid: "082feed59e4143bfaa3839bade529fb1"
format_stamp: "Formatted at 2026-09-24 04:59:09 on dist-test-slave-70st"
server_key: "fbea1c649036ca49d00e5dbf71dba37c"
server_key_iv: "2cccd3592588be2f6e0adc87494a5012"
server_key_version: "encryptionkey@0"
I20260924 04:59:09.722277 3406 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 04:59:09.722997 3421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:09.723162 3406 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 04:59:09.723240 3406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225929848825-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225929848825-392-0/minicluster-data/master-0/wal
uuid: "082feed59e4143bfaa3839bade529fb1"
format_stamp: "Formatted at 2026-09-24 04:59:09 on dist-test-slave-70st"
server_key: "fbea1c649036ca49d00e5dbf71dba37c"
server_key_iv: "2cccd3592588be2f6e0adc87494a5012"
server_key_version: "encryptionkey@0"
I20260924 04:59:09.723306 3406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225929848825-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225929848825-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225929848825-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:59:09.749001 3406 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:59:09.750759 3406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:59:09.750880 3406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:59:09.754479 3406 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43219
I20260924 04:59:09.754562 3473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43219 every 8 connection(s)
I20260924 04:59:09.754841 3406 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:59:09.755793 3474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:59:09.758023 3474 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 082feed59e4143bfaa3839bade529fb1: Bootstrap starting.
I20260924 04:59:09.758550 3474 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 082feed59e4143bfaa3839bade529fb1: Neither blocks nor log segments found. Creating new log.
I20260924 04:59:09.758802 3474 log.cc:824] T 00000000000000000000000000000000 P 082feed59e4143bfaa3839bade529fb1: Log is configured to *not* fsync() on all Append() calls
I20260924 04:59:09.759415 3474 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 082feed59e4143bfaa3839bade529fb1: No bootstrap required, opened a new log
I20260924 04:59:09.760929 3474 raft_consensus.cc:348] T 00000000000000000000000000000000 P 082feed59e4143bfaa3839bade529fb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "082feed59e4143bfaa3839bade529fb1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43219 } }
I20260924 04:59:09.761063 3474 raft_consensus.cc:374] T 00000000000000000000000000000000 P 082feed59e4143bfaa3839bade529fb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:59:09.761093 3474 raft_consensus.cc:729] T 00000000000000000000000000000000 P 082feed59e4143bfaa3839bade529fb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 082feed59e4143bfaa3839bade529fb1, State: Initialized, Role: FOLLOWER
I20260924 04:59:09.761225 3474 consensus_queue.cc:260] T 00000000000000000000000000000000 P 082feed59e4143bfaa3839bade529fb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "082feed59e4143bfaa3839bade529fb1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43219 } }
I20260924 04:59:09.761287 3474 raft_consensus.cc:388] T 00000000000000000000000000000000 P 082feed59e4143bfaa3839bade529fb1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:59:09.761308 3474 raft_consensus.cc:482] T 00000000000000000000000000000000 P 082feed59e4143bfaa3839bade529fb1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:59:09.761354 3474 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 082feed59e4143bfaa3839bade529fb1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:59:09.761499 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 3406
I20260924 04:59:09.761590 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225929848825-392-0/minicluster-data/master-0/wal/instance
I20260924 04:59:09.761742 392 external_mini_cluster.cc:1428] Setting key d1c0364eba1ce063fa2477955bf18956
I20260924 04:59:09.762324 3474 raft_consensus.cc:504] T 00000000000000000000000000000000 P 082feed59e4143bfaa3839bade529fb1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "082feed59e4143bfaa3839bade529fb1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43219 } }
I20260924 04:59:09.762432 3474 leader_election.cc:302] T 00000000000000000000000000000000 P 082feed59e4143bfaa3839bade529fb1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 082feed59e4143bfaa3839bade529fb1; no voters:
I20260924 04:59:09.762579 3474 leader_election.cc:288] T 00000000000000000000000000000000 P 082feed59e4143bfaa3839bade529fb1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:59:09.762599 3479 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 082feed59e4143bfaa3839bade529fb1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:59:09.762698 3479 raft_consensus.cc:686] T 00000000000000000000000000000000 P 082feed59e4143bfaa3839bade529fb1 [term 1 LEADER]: Becoming Leader. State: Replica: 082feed59e4143bfaa3839bade529fb1, State: Running, Role: LEADER
I20260924 04:59:09.762836 3474 sys_catalog.cc:564] T 00000000000000000000000000000000 P 082feed59e4143bfaa3839bade529fb1 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:59:09.762827 3479 consensus_queue.cc:237] T 00000000000000000000000000000000 P 082feed59e4143bfaa3839bade529fb1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "082feed59e4143bfaa3839bade529fb1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43219 } }
I20260924 04:59:09.763139 3481 sys_catalog.cc:455] T 00000000000000000000000000000000 P 082feed59e4143bfaa3839bade529fb1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 082feed59e4143bfaa3839bade529fb1. Latest consensus state: current_term: 1 leader_uuid: "082feed59e4143bfaa3839bade529fb1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "082feed59e4143bfaa3839bade529fb1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43219 } } }
I20260924 04:59:09.763351 3481 sys_catalog.cc:458] T 00000000000000000000000000000000 P 082feed59e4143bfaa3839bade529fb1 [sys.catalog]: This master's current role is: LEADER
I20260924 04:59:09.763294 3480 sys_catalog.cc:455] T 00000000000000000000000000000000 P 082feed59e4143bfaa3839bade529fb1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "082feed59e4143bfaa3839bade529fb1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "082feed59e4143bfaa3839bade529fb1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43219 } } }
I20260924 04:59:09.763571 3480 sys_catalog.cc:458] T 00000000000000000000000000000000 P 082feed59e4143bfaa3839bade529fb1 [sys.catalog]: This master's current role is: LEADER
I20260924 04:59:09.763787 3483 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:59:09.764272 3483 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:59:09.765957 3483 catalog_manager.cc:1347] Generated new cluster ID: f1e7e973833a481989f3a61192dbf0b2
I20260924 04:59:09.766011 3483 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 04:59:09 dist-test-slave-70st krb5kdc[3393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225949, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:59:09.771489 3478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:09.762180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49314 (local address 127.0.98.62:43219)
0924 04:59:09.762344 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:09.762353 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:09.762379 (+ 26us) server_negotiation.cc:407] Connection header received
0924 04:59:09.762496 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:09.762503 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:09.762555 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:09.762723 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:09.763423 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:09.764768 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:09.765776 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:09.766289 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:09.766386 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:09.768563 (+ 2177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:09.768566 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:09.768567 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:59:09.768579 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:09.770097 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:09.770431 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:09.770433 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:09.770434 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:09.770530 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:09.770702 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:09.770703 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:09.770703 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:09.770930 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:09.771084 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:09.771159 (+ 75us) server_negotiation.cc:299] Negotiation successful
0924 04:59:09.771293 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":77,"threads_started":1}
I20260924 04:59:09.772334 3483 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:59:09.773015 3483 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:59:09.788203 3483 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 082feed59e4143bfaa3839bade529fb1: Generated new TSK 0
I20260924 04:59:09.788457 3483 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225929848825-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225929848825-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:59:09.820091 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225929848825-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225929848825-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225929848825-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225929848825-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43219
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225929848825-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:59:09.899930 3500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:59:09.900130 3500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:59:09.900161 3500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:59:09.901311 3500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:59:09.901352 3500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:59:09.901387 3500 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:59:09.902824 3500 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225929848825-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225929848825-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225929848825-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225929848825-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3500
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225929848825-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 04:59:09.903047 3500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:59:09.903249 3500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:59:09.908325 3508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:59:09.908349 3505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:59:09.908349 3506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:59:09.908813 3500 server_base.cc:1034] running on GCE node
I20260924 04:59:09.909010 3500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:59:09.909209 3500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:59:09.910343 3500 hybrid_clock.cc:648] HybridClock initialized: now 1790225949910332 us; error 25 us; skew 500 ppm
Sep 24 04:59:09 dist-test-slave-70st krb5kdc[3393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225949, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:59:09.912619 3500 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:59:09.913036 3500 webserver.cc:460] Webserver started at http://127.0.98.1:42313/ using document root <none> and password file <none>
I20260924 04:59:09.913221 3500 fs_manager.cc:359] Metadata directory not provided
I20260924 04:59:09.913264 3500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:59:09.913379 3500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:59:09.914448 3500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225929848825-392-0/minicluster-data/ts-0/data/instance:
uuid: "22eab8a2de6e47269b472dad90d9b357"
format_stamp: "Formatted at 2026-09-24 04:59:09 on dist-test-slave-70st"
server_key: "b2f2cf6d9a6a1cde2f8b1833f5e2842a"
server_key_iv: "475f51ed3f040728bc0a54c04512abe1"
server_key_version: "encryptionkey@0"
I20260924 04:59:09.914778 3500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225929848825-392-0/minicluster-data/ts-0/wal/instance:
uuid: "22eab8a2de6e47269b472dad90d9b357"
format_stamp: "Formatted at 2026-09-24 04:59:09 on dist-test-slave-70st"
server_key: "b2f2cf6d9a6a1cde2f8b1833f5e2842a"
server_key_iv: "475f51ed3f040728bc0a54c04512abe1"
server_key_version: "encryptionkey@0"
I20260924 04:59:09.915953 3500 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 04:59:09.916553 3515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:09.916677 3500 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 04:59:09.916767 3500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225929848825-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225929848825-392-0/minicluster-data/ts-0/wal
uuid: "22eab8a2de6e47269b472dad90d9b357"
format_stamp: "Formatted at 2026-09-24 04:59:09 on dist-test-slave-70st"
server_key: "b2f2cf6d9a6a1cde2f8b1833f5e2842a"
server_key_iv: "475f51ed3f040728bc0a54c04512abe1"
server_key_version: "encryptionkey@0"
I20260924 04:59:09.916824 3500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225929848825-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225929848825-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225929848825-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:59:09.927963 3500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:59:09.929705 3500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:59:09.929847 3500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:59:09.930049 3500 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:59:09.930331 3500 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:59:09.930363 3500 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:09.930392 3500 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:59:09.930413 3500 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:09.935657 3500 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34807
I20260924 04:59:09.935698 3628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34807 every 8 connection(s)
I20260924 04:59:09.935997 3500 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225929848825-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 04:59:09 dist-test-slave-70st krb5kdc[3393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225949, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:59:09.944721 3478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:09.936693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50437 (local address 127.0.98.62:43219)
0924 04:59:09.936751 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:09.936753 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:09.936936 (+ 183us) server_negotiation.cc:407] Connection header received
0924 04:59:09.937457 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:09.937459 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:09.937475 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:09.937528 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:09.938453 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:09.938957 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:09.939915 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:09.940120 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:09.940174 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:09.942300 (+ 2126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:09.942302 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:09.942302 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:59:09.942310 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:09.943488 (+ 1178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:09.943839 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:09.943840 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:09.943841 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:09.943897 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:09.944107 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:09.944108 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:09.944108 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:09.944222 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:09.944276 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:09.944536 (+ 260us) server_negotiation.cc:299] Negotiation successful
0924 04:59:09.944628 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 04:59:09.944880 3631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:09.936656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43219 (local address 127.0.98.1:50437)
0924 04:59:09.936813 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:09.936847 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:09.937331 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:09.937601 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:09.937605 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:09.938163 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:09.938356 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:09.938361 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:09.939040 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:09.939041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:09.939831 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:09.939834 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:09.940188 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:09.940189 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:09.940301 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:09.940802 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:09.940809 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:09.942218 (+ 1409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:59:09.943558 (+ 1340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:09.943565 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:09.943568 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:09.943776 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:09.943951 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:09.943952 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:09.943952 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:09.944054 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:09.944272 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:59:09.944279 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:59:09.944417 (+ 138us) client_negotiation.cc:769] Sending connection context
0924 04:59:09.944507 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 04:59:09.944661 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":77,"threads_started":1}
I20260924 04:59:09.944970 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 3500
I20260924 04:59:09.945063 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225929848825-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:59:09.945236 392 external_mini_cluster.cc:1428] Setting key 98d8e547b04036f405a13219dfc8ae00
I20260924 04:59:09.945336 3629 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43219
I20260924 04:59:09.945465 3629 heartbeater.cc:464] Registering TS with master...
I20260924 04:59:09.945649 3629 heartbeater.cc:511] Master 127.0.98.62:43219 requested a full tablet report, sending...
I20260924 04:59:09.946149 3438 ts_manager.cc:194] Registered new tserver with Master: 22eab8a2de6e47269b472dad90d9b357 (127.0.98.1:34807)
I20260924 04:59:09.946748 3438 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:50437
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225929848825-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225929848825-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:59:09.967953 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225929848825-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225929848825-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225929848825-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225929848825-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43219
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225929848825-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:59:10.045693 3634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:59:10.045908 3634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:59:10.045934 3634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:59:10.047073 3634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:59:10.047108 3634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:59:10.047133 3634 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:59:10.048506 3634 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225929848825-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225929848825-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225929848825-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225929848825-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3634
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225929848825-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 04:59:10.048693 3634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:59:10.048867 3634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:59:10.053362 3639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:59:10.053355 3642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:59:10.053637 3640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:59:10.053764 3634 server_base.cc:1034] running on GCE node
I20260924 04:59:10.053952 3634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:59:10.054167 3634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:59:10.055361 3634 hybrid_clock.cc:648] HybridClock initialized: now 1790225950055341 us; error 34 us; skew 500 ppm
Sep 24 04:59:10 dist-test-slave-70st krb5kdc[3393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225950, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:59:10.057891 3634 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:59:10.058280 3634 webserver.cc:460] Webserver started at http://127.0.98.2:36485/ using document root <none> and password file <none>
I20260924 04:59:10.058466 3634 fs_manager.cc:359] Metadata directory not provided
I20260924 04:59:10.058506 3634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:59:10.058598 3634 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:59:10.059587 3634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225929848825-392-0/minicluster-data/ts-1/data/instance:
uuid: "1850f76cab924bd6bec7185b320e4de2"
format_stamp: "Formatted at 2026-09-24 04:59:10 on dist-test-slave-70st"
server_key: "e0ac8a6e6948ea75024002bb0e352d5f"
server_key_iv: "10a2605c9a5dbfd9372c9b759a50ac72"
server_key_version: "encryptionkey@0"
I20260924 04:59:10.059871 3634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225929848825-392-0/minicluster-data/ts-1/wal/instance:
uuid: "1850f76cab924bd6bec7185b320e4de2"
format_stamp: "Formatted at 2026-09-24 04:59:10 on dist-test-slave-70st"
server_key: "e0ac8a6e6948ea75024002bb0e352d5f"
server_key_iv: "10a2605c9a5dbfd9372c9b759a50ac72"
server_key_version: "encryptionkey@0"
I20260924 04:59:10.061077 3634 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 04:59:10.061830 3649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:10.061995 3634 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 04:59:10.062088 3634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225929848825-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225929848825-392-0/minicluster-data/ts-1/wal
uuid: "1850f76cab924bd6bec7185b320e4de2"
format_stamp: "Formatted at 2026-09-24 04:59:10 on dist-test-slave-70st"
server_key: "e0ac8a6e6948ea75024002bb0e352d5f"
server_key_iv: "10a2605c9a5dbfd9372c9b759a50ac72"
server_key_version: "encryptionkey@0"
I20260924 04:59:10.062153 3634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225929848825-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225929848825-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225929848825-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:59:10.072958 3634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:59:10.075013 3634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:59:10.075119 3634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:59:10.075289 3634 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:59:10.075529 3634 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:59:10.075557 3634 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:10.075583 3634 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:59:10.075599 3634 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:10.080718 3634 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35259
I20260924 04:59:10.080762 3762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35259 every 8 connection(s)
I20260924 04:59:10.081029 3634 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225929848825-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:59:10.082098 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 3634
I20260924 04:59:10.082171 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225929848825-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:59:10.082316 392 external_mini_cluster.cc:1428] Setting key ca86a0444362c05f286a2891241f0775
Sep 24 04:59:10 dist-test-slave-70st krb5kdc[3393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225950, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 04:59:10.089916 3478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:10.081668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35531 (local address 127.0.98.62:43219)
0924 04:59:10.081742 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:10.081745 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:10.081884 (+ 139us) server_negotiation.cc:407] Connection header received
0924 04:59:10.082439 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:10.082441 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:10.082452 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:10.082502 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:10.083384 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:10.083869 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:10.084805 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:10.084995 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:10.085049 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:10.087222 (+ 2173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:10.087224 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:10.087224 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:59:10.087231 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:10.088635 (+ 1404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:10.088987 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:10.088989 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:10.088989 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:10.089047 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:10.089243 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:10.089244 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:10.089245 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:10.089378 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:10.089482 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:10.089722 (+ 240us) server_negotiation.cc:299] Negotiation successful
0924 04:59:10.089812 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 04:59:10.089986 3765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:10.081644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43219 (local address 127.0.98.2:35531)
0924 04:59:10.081768 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:10.081801 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:10.082322 (+ 521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:10.082573 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:10.082577 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:10.083120 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:10.083299 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:10.083304 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:10.083944 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:10.083945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:10.084730 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:10.084732 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:10.085061 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:10.085061 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:10.085178 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:10.085661 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:10.085671 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:10.087126 (+ 1455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:59:10.088708 (+ 1582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:10.088713 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:10.088714 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:10.088918 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:10.089106 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:10.089106 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:10.089106 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:10.089189 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:10.089472 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:59:10.089477 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:59:10.089600 (+ 123us) client_negotiation.cc:769] Sending connection context
0924 04:59:10.089685 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 04:59:10.089825 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":66,"threads_started":1}
I20260924 04:59:10.090386 3763 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43219
I20260924 04:59:10.090466 3763 heartbeater.cc:464] Registering TS with master...
I20260924 04:59:10.090641 3763 heartbeater.cc:511] Master 127.0.98.62:43219 requested a full tablet report, sending...
I20260924 04:59:10.090981 3438 ts_manager.cc:194] Registered new tserver with Master: 1850f76cab924bd6bec7185b320e4de2 (127.0.98.2:35259)
I20260924 04:59:10.091331 3438 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35531
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225929848825-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225929848825-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:59:10.105145 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225929848825-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225929848825-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225929848825-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225929848825-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43219
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225929848825-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:59:10.182832 3768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:59:10.183028 3768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:59:10.183049 3768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:59:10.184170 3768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:59:10.184207 3768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:59:10.184231 3768 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:59:10.185633 3768 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225929848825-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225929848825-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225929848825-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225929848825-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3768
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225929848825-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 04:59:10.185854 3768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:59:10.186080 3768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:59:10.190675 3768 server_base.cc:1034] running on GCE node
W20260924 04:59:10.190724 3774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:59:10.190649 3773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:59:10.190659 3776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:59:10.190995 3768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:59:10.191216 3768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:59:10.192353 3768 hybrid_clock.cc:648] HybridClock initialized: now 1790225950192336 us; error 29 us; skew 500 ppm
Sep 24 04:59:10 dist-test-slave-70st krb5kdc[3393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225950, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:59:10.194645 3768 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:59:10.195043 3768 webserver.cc:460] Webserver started at http://127.0.98.3:45099/ using document root <none> and password file <none>
I20260924 04:59:10.195242 3768 fs_manager.cc:359] Metadata directory not provided
I20260924 04:59:10.195290 3768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:59:10.195407 3768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:59:10.196391 3768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225929848825-392-0/minicluster-data/ts-2/data/instance:
uuid: "fc8e5c871d69483d8871db1cb192f885"
format_stamp: "Formatted at 2026-09-24 04:59:10 on dist-test-slave-70st"
server_key: "107af6ee68abc7ef2b6cf7f5b86e8fd6"
server_key_iv: "e8571a14ce27c01d6f1b16d9eecb3f3b"
server_key_version: "encryptionkey@0"
I20260924 04:59:10.196694 3768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225929848825-392-0/minicluster-data/ts-2/wal/instance:
uuid: "fc8e5c871d69483d8871db1cb192f885"
format_stamp: "Formatted at 2026-09-24 04:59:10 on dist-test-slave-70st"
server_key: "107af6ee68abc7ef2b6cf7f5b86e8fd6"
server_key_iv: "e8571a14ce27c01d6f1b16d9eecb3f3b"
server_key_version: "encryptionkey@0"
I20260924 04:59:10.197923 3768 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 04:59:10.198547 3783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:10.198697 3768 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 04:59:10.198781 3768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225929848825-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225929848825-392-0/minicluster-data/ts-2/wal
uuid: "fc8e5c871d69483d8871db1cb192f885"
format_stamp: "Formatted at 2026-09-24 04:59:10 on dist-test-slave-70st"
server_key: "107af6ee68abc7ef2b6cf7f5b86e8fd6"
server_key_iv: "e8571a14ce27c01d6f1b16d9eecb3f3b"
server_key_version: "encryptionkey@0"
I20260924 04:59:10.198837 3768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225929848825-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225929848825-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225929848825-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:59:10.259913 3768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:59:10.261823 3768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:59:10.261952 3768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:59:10.262159 3768 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:59:10.262470 3768 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:59:10.262506 3768 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:10.262534 3768 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:59:10.262552 3768 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:10.268038 3768 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40175
I20260924 04:59:10.268085 3896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40175 every 8 connection(s)
I20260924 04:59:10.268374 3768 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225929848825-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:59:10.270164 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 3768
I20260924 04:59:10.270239 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790225929848825-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:59:10.270391 392 external_mini_cluster.cc:1428] Setting key 3a50dcc44281edc50146dddf9244a5fc
Sep 24 04:59:10 dist-test-slave-70st krb5kdc[3393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225950, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:59:10.276826 3478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:10.268976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41149 (local address 127.0.98.62:43219)
0924 04:59:10.269043 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:10.269045 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:10.269232 (+ 187us) server_negotiation.cc:407] Connection header received
0924 04:59:10.269795 (+ 563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:10.269796 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:10.269808 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:10.269880 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:10.270831 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:10.271324 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:10.272291 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:10.272491 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:10.272545 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:10.274516 (+ 1971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:10.274517 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:10.274518 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:59:10.274525 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:10.275634 (+ 1109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:10.275987 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:10.275988 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:10.275988 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:10.276041 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:10.276247 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:10.276248 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:10.276249 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:10.276358 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:10.276412 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:10.276658 (+ 246us) server_negotiation.cc:299] Negotiation successful
0924 04:59:10.276735 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 04:59:10.277019 3899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:10.268965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43219 (local address 127.0.98.3:41149)
0924 04:59:10.269120 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:10.269155 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:10.269685 (+ 530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:10.269959 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:10.269962 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:10.270545 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:10.270746 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:10.270751 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:10.271406 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:10.271426 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:10.272214 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:10.272216 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:10.272556 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:10.272557 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:10.272673 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:10.273133 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:10.273141 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:10.274431 (+ 1290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:59:10.275706 (+ 1275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:10.275712 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:10.275715 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:10.275923 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:10.276092 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:10.276092 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:10.276093 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:10.276187 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:10.276410 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:59:10.276416 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:59:10.276546 (+ 130us) client_negotiation.cc:769] Sending connection context
0924 04:59:10.276632 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 04:59:10.276803 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":79,"threads_started":1}
I20260924 04:59:10.277443 3897 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43219
I20260924 04:59:10.277534 3897 heartbeater.cc:464] Registering TS with master...
I20260924 04:59:10.277673 3897 heartbeater.cc:511] Master 127.0.98.62:43219 requested a full tablet report, sending...
I20260924 04:59:10.277954 3438 ts_manager.cc:194] Registered new tserver with Master: fc8e5c871d69483d8871db1cb192f885 (127.0.98.3:40175)
I20260924 04:59:10.278322 3438 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:41149
I20260924 04:59:10.281854 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 04:59:10 dist-test-slave-70st krb5kdc[3393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225950, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 24 04:59:10 dist-test-slave-70st krb5kdc[3393](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225950, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:59:10.302261 3478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:10.295924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49316 (local address 127.0.98.62:43219)
0924 04:59:10.296009 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:10.296012 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:10.296077 (+ 65us) server_negotiation.cc:407] Connection header received
0924 04:59:10.296132 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:10.296133 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:10.296143 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:10.296192 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:10.296692 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:10.297184 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:10.298006 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:10.298193 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:10.298250 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:10.300263 (+ 2013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:10.300265 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:10.300265 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:59:10.300272 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:10.301359 (+ 1087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:10.301661 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:10.301662 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:10.301662 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:10.301708 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:10.301846 (+ 138us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:10.301846 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:10.301847 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:10.301985 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:10.302051 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:10.302125 (+ 74us) server_negotiation.cc:299] Negotiation successful
0924 04:59:10.302192 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260924 04:59:10.302484 3438 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:49316
I20260924 04:59:10.302829 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 3500
I20260924 04:59:10.306861 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 3634
I20260924 04:59:10.310570 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 3768
I20260924 04:59:10.313923 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 3406
2026-09-24T04:59:10Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1767 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225929848825-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:59:10 dist-test-slave-70st krb5kdc[3909](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:59:10 dist-test-slave-70st krb5kdc[3909](info): set up 2 sockets
Sep 24 04:59:10 dist-test-slave-70st krb5kdc[3909](info): commencing operation
krb5kdc: starting...
W20260924 04:59:11.257182 392 mini_kdc.cc:121] Time spent starting KDC: real 0.925s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225929848825-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225929848825-392-0/krb5kdc/test-user.keytab.
Sep 24 04:59:11 dist-test-slave-70st krb5kdc[3909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225951, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:59:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:59:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:59:11.367611 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225929848825-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225929848825-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225929848825-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34661
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34661
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:59:11.445335 3922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:59:11.445572 3922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:59:11.445602 3922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:59:11.446784 3922 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:59:11.446830 3922 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:59:11.446848 3922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:59:11.446864 3922 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:59:11.446879 3922 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:59:11.448325 3922 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225929848825-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225929848825-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34661
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34661
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3922
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225929848825-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 04:59:11.448561 3922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:59:11.448767 3922 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:59:11.453451 3927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:59:11.453444 3928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:59:11.453518 3930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:59:11.453593 3922 server_base.cc:1034] running on GCE node
I20260924 04:59:11.453912 3922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:59:11.454205 3922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:59:11.455355 3922 hybrid_clock.cc:648] HybridClock initialized: now 1790225951455338 us; error 27 us; skew 500 ppm
Sep 24 04:59:11 dist-test-slave-70st krb5kdc[3909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225951, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:59:11.457787 3922 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:59:11.458173 3922 webserver.cc:460] Webserver started at http://127.0.98.62:37011/ using document root <none> and password file <none>
I20260924 04:59:11.458344 3922 fs_manager.cc:359] Metadata directory not provided
I20260924 04:59:11.458384 3922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:59:11.458472 3922 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:59:11.459470 3922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225929848825-392-0/minicluster-data/master-0/data/instance:
uuid: "5ec839cfd0b04dacbd89fd5825a13b0e"
format_stamp: "Formatted at 2026-09-24 04:59:11 on dist-test-slave-70st"
server_key: "e106ec3b9d8eade904650a610c93b1ce"
server_key_iv: "d476cce2ca00c972401dd40b2f9950c6"
server_key_version: "encryptionkey@0"
I20260924 04:59:11.459758 3922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225929848825-392-0/minicluster-data/master-0/wal/instance:
uuid: "5ec839cfd0b04dacbd89fd5825a13b0e"
format_stamp: "Formatted at 2026-09-24 04:59:11 on dist-test-slave-70st"
server_key: "e106ec3b9d8eade904650a610c93b1ce"
server_key_iv: "d476cce2ca00c972401dd40b2f9950c6"
server_key_version: "encryptionkey@0"
I20260924 04:59:11.460942 3922 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 04:59:11.461596 3937 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:11.461743 3922 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:11.461826 3922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225929848825-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225929848825-392-0/minicluster-data/master-0/wal
uuid: "5ec839cfd0b04dacbd89fd5825a13b0e"
format_stamp: "Formatted at 2026-09-24 04:59:11 on dist-test-slave-70st"
server_key: "e106ec3b9d8eade904650a610c93b1ce"
server_key_iv: "d476cce2ca00c972401dd40b2f9950c6"
server_key_version: "encryptionkey@0"
I20260924 04:59:11.461880 3922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225929848825-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225929848825-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225929848825-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:59:11.469931 3922 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:59:11.471660 3922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:59:11.471774 3922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:59:11.475379 3922 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34661
I20260924 04:59:11.475440 3989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34661 every 8 connection(s)
I20260924 04:59:11.475768 3922 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:59:11.476585 3990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:59:11.478860 3990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ec839cfd0b04dacbd89fd5825a13b0e: Bootstrap starting.
I20260924 04:59:11.479409 3990 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5ec839cfd0b04dacbd89fd5825a13b0e: Neither blocks nor log segments found. Creating new log.
I20260924 04:59:11.479658 3990 log.cc:824] T 00000000000000000000000000000000 P 5ec839cfd0b04dacbd89fd5825a13b0e: Log is configured to *not* fsync() on all Append() calls
I20260924 04:59:11.480355 3990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ec839cfd0b04dacbd89fd5825a13b0e: No bootstrap required, opened a new log
I20260924 04:59:11.481806 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 3922
I20260924 04:59:11.481887 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225929848825-392-0/minicluster-data/master-0/wal/instance
I20260924 04:59:11.482045 392 external_mini_cluster.cc:1428] Setting key cb2cc611b7a487c32e4f204b26b99be4
I20260924 04:59:11.482007 3990 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ec839cfd0b04dacbd89fd5825a13b0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ec839cfd0b04dacbd89fd5825a13b0e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34661 } }
I20260924 04:59:11.482164 3990 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5ec839cfd0b04dacbd89fd5825a13b0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:59:11.482204 3990 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ec839cfd0b04dacbd89fd5825a13b0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ec839cfd0b04dacbd89fd5825a13b0e, State: Initialized, Role: FOLLOWER
I20260924 04:59:11.482342 3990 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ec839cfd0b04dacbd89fd5825a13b0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ec839cfd0b04dacbd89fd5825a13b0e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34661 } }
I20260924 04:59:11.482445 3990 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5ec839cfd0b04dacbd89fd5825a13b0e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:59:11.482487 3990 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5ec839cfd0b04dacbd89fd5825a13b0e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:59:11.482559 3990 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ec839cfd0b04dacbd89fd5825a13b0e [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:59:11.483141 3990 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5ec839cfd0b04dacbd89fd5825a13b0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ec839cfd0b04dacbd89fd5825a13b0e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34661 } }
I20260924 04:59:11.483228 3990 leader_election.cc:302] T 00000000000000000000000000000000 P 5ec839cfd0b04dacbd89fd5825a13b0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5ec839cfd0b04dacbd89fd5825a13b0e; no voters:
I20260924 04:59:11.483384 3990 leader_election.cc:288] T 00000000000000000000000000000000 P 5ec839cfd0b04dacbd89fd5825a13b0e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:59:11.483417 3995 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5ec839cfd0b04dacbd89fd5825a13b0e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:59:11.483513 3995 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5ec839cfd0b04dacbd89fd5825a13b0e [term 1 LEADER]: Becoming Leader. State: Replica: 5ec839cfd0b04dacbd89fd5825a13b0e, State: Running, Role: LEADER
I20260924 04:59:11.483592 3995 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5ec839cfd0b04dacbd89fd5825a13b0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ec839cfd0b04dacbd89fd5825a13b0e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34661 } }
I20260924 04:59:11.483649 3990 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ec839cfd0b04dacbd89fd5825a13b0e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:59:11.483935 3995 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ec839cfd0b04dacbd89fd5825a13b0e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5ec839cfd0b04dacbd89fd5825a13b0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ec839cfd0b04dacbd89fd5825a13b0e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34661 } } }
I20260924 04:59:11.484090 3995 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ec839cfd0b04dacbd89fd5825a13b0e [sys.catalog]: This master's current role is: LEADER
I20260924 04:59:11.484253 3995 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ec839cfd0b04dacbd89fd5825a13b0e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ec839cfd0b04dacbd89fd5825a13b0e. Latest consensus state: current_term: 1 leader_uuid: "5ec839cfd0b04dacbd89fd5825a13b0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ec839cfd0b04dacbd89fd5825a13b0e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34661 } } }
I20260924 04:59:11.484359 3995 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ec839cfd0b04dacbd89fd5825a13b0e [sys.catalog]: This master's current role is: LEADER
I20260924 04:59:11.484686 4003 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:59:11.485234 4003 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:59:11.486826 4003 catalog_manager.cc:1347] Generated new cluster ID: 783d7d3192b04640944da4c3e8a0d9fd
I20260924 04:59:11.486877 4003 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 04:59:11 dist-test-slave-70st krb5kdc[3909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225951, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:59:11.493410 3994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:11.482482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53780 (local address 127.0.98.62:34661)
0924 04:59:11.482737 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:11.482745 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:11.482772 (+ 27us) server_negotiation.cc:407] Connection header received
0924 04:59:11.483096 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:11.483103 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:11.483149 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:11.483305 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:11.484119 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:11.485513 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:11.486395 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:11.486991 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:11.487103 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:11.489812 (+ 2709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:11.489816 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:11.489817 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:59:11.489831 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:11.491738 (+ 1907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:11.492163 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:11.492166 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:11.492166 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:11.492286 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:11.492482 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:11.492483 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:11.492483 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:11.492771 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:11.492921 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:11.493037 (+ 116us) server_negotiation.cc:299] Negotiation successful
0924 04:59:11.493179 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":76,"threads_started":1}
I20260924 04:59:11.493589 4003 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:59:11.494122 4003 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:59:11.497912 4003 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5ec839cfd0b04dacbd89fd5825a13b0e: Generated new TSK 0
I20260924 04:59:11.498080 4003 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225929848825-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225929848825-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:59:11.526425 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225929848825-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225929848825-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225929848825-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225929848825-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34661
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225929848825-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:59:11.605130 4016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:59:11.605337 4016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:59:11.605368 4016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:59:11.606539 4016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:59:11.606578 4016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:59:11.606613 4016 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:59:11.608047 4016 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225929848825-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225929848825-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225929848825-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225929848825-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34661
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4016
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225929848825-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 04:59:11.608275 4016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:59:11.608479 4016 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:59:11.613333 4022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:59:11.613373 4024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:59:11.613348 4021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:59:11.613547 4016 server_base.cc:1034] running on GCE node
I20260924 04:59:11.613759 4016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:59:11.613976 4016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:59:11.615128 4016 hybrid_clock.cc:648] HybridClock initialized: now 1790225951615115 us; error 25 us; skew 500 ppm
Sep 24 04:59:11 dist-test-slave-70st krb5kdc[3909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225951, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:59:11.617544 4016 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:59:11.617996 4016 webserver.cc:460] Webserver started at http://127.0.98.1:36335/ using document root <none> and password file <none>
I20260924 04:59:11.618202 4016 fs_manager.cc:359] Metadata directory not provided
I20260924 04:59:11.618247 4016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:59:11.618369 4016 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:59:11.619382 4016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225929848825-392-0/minicluster-data/ts-0/data/instance:
uuid: "5ce9fdf4b9ee4ecd8acb1628614a7e84"
format_stamp: "Formatted at 2026-09-24 04:59:11 on dist-test-slave-70st"
server_key: "4d4a9f06e7f60b32b3dbdaaf310e9c53"
server_key_iv: "a432a37b86e6bb3c2226a9551122112e"
server_key_version: "encryptionkey@0"
I20260924 04:59:11.619688 4016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225929848825-392-0/minicluster-data/ts-0/wal/instance:
uuid: "5ce9fdf4b9ee4ecd8acb1628614a7e84"
format_stamp: "Formatted at 2026-09-24 04:59:11 on dist-test-slave-70st"
server_key: "4d4a9f06e7f60b32b3dbdaaf310e9c53"
server_key_iv: "a432a37b86e6bb3c2226a9551122112e"
server_key_version: "encryptionkey@0"
I20260924 04:59:11.620869 4016 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 04:59:11.621507 4031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:11.621662 4016 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:11.621743 4016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225929848825-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225929848825-392-0/minicluster-data/ts-0/wal
uuid: "5ce9fdf4b9ee4ecd8acb1628614a7e84"
format_stamp: "Formatted at 2026-09-24 04:59:11 on dist-test-slave-70st"
server_key: "4d4a9f06e7f60b32b3dbdaaf310e9c53"
server_key_iv: "a432a37b86e6bb3c2226a9551122112e"
server_key_version: "encryptionkey@0"
I20260924 04:59:11.621810 4016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225929848825-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225929848825-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225929848825-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:59:11.634789 4016 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:59:11.636575 4016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:59:11.636711 4016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:59:11.636922 4016 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:59:11.637218 4016 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:59:11.637251 4016 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:11.637290 4016 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:59:11.637313 4016 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:11.642820 4016 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39765
I20260924 04:59:11.642876 4144 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39765 every 8 connection(s)
I20260924 04:59:11.643159 4016 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225929848825-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 04:59:11 dist-test-slave-70st krb5kdc[3909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225951, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:59:11.651190 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 4016
I20260924 04:59:11.651285 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225929848825-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:59:11.651492 392 external_mini_cluster.cc:1428] Setting key 6760b52ccddc211899f1f0851b24b679
I20260924 04:59:11.651830 3994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:11.643834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47059 (local address 127.0.98.62:34661)
0924 04:59:11.643885 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:11.643887 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:11.644057 (+ 170us) server_negotiation.cc:407] Connection header received
0924 04:59:11.645000 (+ 943us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:11.645001 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:11.645017 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:11.645067 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:11.645702 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:11.646204 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:11.647160 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:11.647371 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:11.647426 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:11.649389 (+ 1963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:11.649391 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:11.649391 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:59:11.649401 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:11.650572 (+ 1171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:11.650914 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:11.650915 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:11.650915 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:11.650965 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:11.651159 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:11.651160 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:11.651161 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:11.651263 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:11.651316 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:11.651642 (+ 326us) server_negotiation.cc:299] Negotiation successful
0924 04:59:11.651722 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 04:59:11.652004 4147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:11.643806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34661 (local address 127.0.98.1:47059)
0924 04:59:11.643944 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:11.643979 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:11.644899 (+ 920us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:11.645134 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:11.645137 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:11.645389 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:11.645616 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:11.645621 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:11.646296 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:11.646298 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:11.647077 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:11.647079 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:11.647436 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:11.647438 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:11.647551 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:11.647961 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:11.647969 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:11.649243 (+ 1274us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:59:11.650643 (+ 1400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:11.650649 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:11.650651 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:11.650852 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:11.651016 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:11.651017 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:11.651017 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:11.651108 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:11.651333 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:59:11.651340 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:59:11.651505 (+ 165us) client_negotiation.cc:769] Sending connection context
0924 04:59:11.651629 (+ 124us) client_negotiation.cc:240] Negotiation successful
0924 04:59:11.651778 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":74,"threads_started":1}
I20260924 04:59:11.652546 4145 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34661
I20260924 04:59:11.652640 4145 heartbeater.cc:464] Registering TS with master...
I20260924 04:59:11.652810 4145 heartbeater.cc:511] Master 127.0.98.62:34661 requested a full tablet report, sending...
I20260924 04:59:11.653298 3954 ts_manager.cc:194] Registered new tserver with Master: 5ce9fdf4b9ee4ecd8acb1628614a7e84 (127.0.98.1:39765)
I20260924 04:59:11.653909 3954 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:47059
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225929848825-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225929848825-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:59:11.675855 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225929848825-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225929848825-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225929848825-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225929848825-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34661
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225929848825-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:59:11.754755 4150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:59:11.755007 4150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:59:11.755039 4150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:59:11.756177 4150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:59:11.756219 4150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:59:11.756254 4150 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:59:11.757732 4150 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225929848825-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225929848825-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225929848825-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225929848825-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34661
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4150
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225929848825-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 04:59:11.757973 4150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:59:11.758188 4150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:59:11.763020 4155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:59:11.763113 4150 server_base.cc:1034] running on GCE node
W20260924 04:59:11.763055 4158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:59:11.763645 4156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:59:11.763865 4150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:59:11.764092 4150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:59:11.765234 4150 hybrid_clock.cc:648] HybridClock initialized: now 1790225951765219 us; error 30 us; skew 500 ppm
Sep 24 04:59:11 dist-test-slave-70st krb5kdc[3909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225951, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:59:11.767715 4150 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:59:11.768149 4150 webserver.cc:460] Webserver started at http://127.0.98.2:36339/ using document root <none> and password file <none>
I20260924 04:59:11.768369 4150 fs_manager.cc:359] Metadata directory not provided
I20260924 04:59:11.768416 4150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:59:11.768540 4150 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:59:11.769619 4150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225929848825-392-0/minicluster-data/ts-1/data/instance:
uuid: "c6f79a6bcdc0487ca5d42c048163b4c7"
format_stamp: "Formatted at 2026-09-24 04:59:11 on dist-test-slave-70st"
server_key: "e36fdb9a1d811d03eafc4282445b5937"
server_key_iv: "c2d3c8177fa8cd342f4ebef1a00267b1"
server_key_version: "encryptionkey@0"
I20260924 04:59:11.769940 4150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225929848825-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c6f79a6bcdc0487ca5d42c048163b4c7"
format_stamp: "Formatted at 2026-09-24 04:59:11 on dist-test-slave-70st"
server_key: "e36fdb9a1d811d03eafc4282445b5937"
server_key_iv: "c2d3c8177fa8cd342f4ebef1a00267b1"
server_key_version: "encryptionkey@0"
I20260924 04:59:11.771162 4150 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 04:59:11.771948 4165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:11.772106 4150 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 04:59:11.772189 4150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225929848825-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225929848825-392-0/minicluster-data/ts-1/wal
uuid: "c6f79a6bcdc0487ca5d42c048163b4c7"
format_stamp: "Formatted at 2026-09-24 04:59:11 on dist-test-slave-70st"
server_key: "e36fdb9a1d811d03eafc4282445b5937"
server_key_iv: "c2d3c8177fa8cd342f4ebef1a00267b1"
server_key_version: "encryptionkey@0"
I20260924 04:59:11.772250 4150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225929848825-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225929848825-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225929848825-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:59:11.781270 4150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:59:11.783257 4150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:59:11.783392 4150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:59:11.783610 4150 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:59:11.783914 4150 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:59:11.783947 4150 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:11.783974 4150 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:59:11.783998 4150 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:11.789592 4150 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46827
I20260924 04:59:11.789647 4278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46827 every 8 connection(s)
I20260924 04:59:11.789958 4150 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225929848825-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:59:11.790234 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 4150
I20260924 04:59:11.790335 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225929848825-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:59:11.790517 392 external_mini_cluster.cc:1428] Setting key c945f1b037ab3729c0d668a86e71731d
Sep 24 04:59:11 dist-test-slave-70st krb5kdc[3909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225951, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 04:59:11.799746 3994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:11.790679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:32985 (local address 127.0.98.62:34661)
0924 04:59:11.790730 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:11.790732 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:11.790986 (+ 254us) server_negotiation.cc:407] Connection header received
0924 04:59:11.791978 (+ 992us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:11.791979 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:11.791991 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:11.792044 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:11.792680 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:11.793268 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:11.794403 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:11.794694 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:11.794772 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:11.797009 (+ 2237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:11.797011 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:11.797011 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:59:11.797019 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:11.798440 (+ 1421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:11.798791 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:11.798792 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:11.798792 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:11.798837 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:11.799125 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:11.799126 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:11.799126 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:11.799233 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:11.799284 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:11.799579 (+ 295us) server_negotiation.cc:299] Negotiation successful
0924 04:59:11.799675 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 04:59:11.799959 4281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:11.790675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34661 (local address 127.0.98.2:32985)
0924 04:59:11.790845 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:11.790901 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:11.791864 (+ 963us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:11.792138 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:11.792147 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:11.792403 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:11.792593 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:11.792597 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:11.793348 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:11.793349 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:11.794306 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:11.794308 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:11.794769 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:11.794770 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:11.794876 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:11.795412 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:11.795420 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:11.796913 (+ 1493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:59:11.798510 (+ 1597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:11.798516 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:11.798518 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:11.798734 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:11.798943 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:11.798944 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:11.798944 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:11.799063 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:11.799321 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:59:11.799327 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:59:11.799460 (+ 133us) client_negotiation.cc:769] Sending connection context
0924 04:59:11.799560 (+ 100us) client_negotiation.cc:240] Negotiation successful
0924 04:59:11.799717 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":85,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260924 04:59:11.800503 4279 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34661
I20260924 04:59:11.800611 4279 heartbeater.cc:464] Registering TS with master...
I20260924 04:59:11.800796 4279 heartbeater.cc:511] Master 127.0.98.62:34661 requested a full tablet report, sending...
I20260924 04:59:11.801185 3954 ts_manager.cc:194] Registered new tserver with Master: c6f79a6bcdc0487ca5d42c048163b4c7 (127.0.98.2:46827)
I20260924 04:59:11.801618 3954 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:32985
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225929848825-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225929848825-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:59:11.814442 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225929848825-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225929848825-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225929848825-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225929848825-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34661
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225929848825-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:59:11.894415 4284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:59:11.894639 4284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:59:11.894670 4284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:59:11.895907 4284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:59:11.895956 4284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:59:11.895992 4284 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:59:11.897571 4284 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225929848825-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225929848825-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225929848825-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225929848825-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34661
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4284
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225929848825-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 04:59:11.897830 4284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:59:11.898056 4284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:59:11.902877 4292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:59:11.903008 4284 server_base.cc:1034] running on GCE node
W20260924 04:59:11.902886 4290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:59:11.902896 4289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:59:11.903283 4284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:59:11.903522 4284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:59:11.904672 4284 hybrid_clock.cc:648] HybridClock initialized: now 1790225951904656 us; error 33 us; skew 500 ppm
Sep 24 04:59:11 dist-test-slave-70st krb5kdc[3909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225951, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:59:11.907090 4284 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:59:11.907486 4284 webserver.cc:460] Webserver started at http://127.0.98.3:39871/ using document root <none> and password file <none>
I20260924 04:59:11.907665 4284 fs_manager.cc:359] Metadata directory not provided
I20260924 04:59:11.907703 4284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:59:11.907793 4284 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:59:11.908855 4284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225929848825-392-0/minicluster-data/ts-2/data/instance:
uuid: "da2a24dd0cad4850b3936b02569c0855"
format_stamp: "Formatted at 2026-09-24 04:59:11 on dist-test-slave-70st"
server_key: "1585ea0728bf0c402990e7facd6c0b71"
server_key_iv: "e7958ea90fa0027155ccc714534aae9c"
server_key_version: "encryptionkey@0"
I20260924 04:59:11.909157 4284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225929848825-392-0/minicluster-data/ts-2/wal/instance:
uuid: "da2a24dd0cad4850b3936b02569c0855"
format_stamp: "Formatted at 2026-09-24 04:59:11 on dist-test-slave-70st"
server_key: "1585ea0728bf0c402990e7facd6c0b71"
server_key_iv: "e7958ea90fa0027155ccc714534aae9c"
server_key_version: "encryptionkey@0"
I20260924 04:59:11.910512 4284 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 04:59:11.911240 4299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:11.911412 4284 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 04:59:11.911494 4284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225929848825-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225929848825-392-0/minicluster-data/ts-2/wal
uuid: "da2a24dd0cad4850b3936b02569c0855"
format_stamp: "Formatted at 2026-09-24 04:59:11 on dist-test-slave-70st"
server_key: "1585ea0728bf0c402990e7facd6c0b71"
server_key_iv: "e7958ea90fa0027155ccc714534aae9c"
server_key_version: "encryptionkey@0"
I20260924 04:59:11.911550 4284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225929848825-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225929848825-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225929848825-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:59:11.935576 4284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:59:11.937810 4284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:59:11.937918 4284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:59:11.938097 4284 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:59:11.938333 4284 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:59:11.938361 4284 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:11.938380 4284 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:59:11.938395 4284 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:11.943610 4284 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44423
I20260924 04:59:11.943658 4412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44423 every 8 connection(s)
I20260924 04:59:11.943953 4284 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225929848825-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:59:11.948978 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 4284
I20260924 04:59:11.949090 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790225929848825-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:59:11.949254 392 external_mini_cluster.cc:1428] Setting key 3fafc02d0295266a03bacdd0e746215b
Sep 24 04:59:11 dist-test-slave-70st krb5kdc[3909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225951, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:59:11.952723 3994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:11.944637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39023 (local address 127.0.98.62:34661)
0924 04:59:11.944678 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:11.944680 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:11.944877 (+ 197us) server_negotiation.cc:407] Connection header received
0924 04:59:11.945853 (+ 976us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:11.945855 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:11.945867 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:11.945910 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:11.946532 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:11.947021 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:11.947982 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:11.948164 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:11.948219 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:11.950453 (+ 2234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:11.950454 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:11.950455 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:59:11.950463 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:11.951566 (+ 1103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:11.951960 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:11.951961 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:11.951961 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:11.952012 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:11.952195 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:11.952195 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:11.952195 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:11.952318 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:11.952364 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:11.952597 (+ 233us) server_negotiation.cc:299] Negotiation successful
0924 04:59:11.952659 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 04:59:11.952921 4415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:11.944604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34661 (local address 127.0.98.3:39023)
0924 04:59:11.944764 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:11.944801 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:11.945755 (+ 954us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:11.945994 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:11.945998 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:11.946250 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:11.946438 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:11.946451 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:11.947101 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:11.947102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:11.947897 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:11.947899 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:11.948231 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:11.948231 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:11.948349 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:11.948848 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:11.948858 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:11.950382 (+ 1524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:59:11.951633 (+ 1251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:11.951639 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:11.951642 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:11.951880 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:11.952059 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:11.952060 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:11.952060 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:11.952154 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:11.952360 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:59:11.952362 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:59:11.952486 (+ 124us) client_negotiation.cc:769] Sending connection context
0924 04:59:11.952578 (+ 92us) client_negotiation.cc:240] Negotiation successful
0924 04:59:11.952698 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":80,"threads_started":1}
I20260924 04:59:11.953375 4413 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34661
I20260924 04:59:11.953537 4413 heartbeater.cc:464] Registering TS with master...
I20260924 04:59:11.953686 4413 heartbeater.cc:511] Master 127.0.98.62:34661 requested a full tablet report, sending...
I20260924 04:59:11.954032 3954 ts_manager.cc:194] Registered new tserver with Master: da2a24dd0cad4850b3936b02569c0855 (127.0.98.3:44423)
I20260924 04:59:11.954418 3954 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:39023
I20260924 04:59:11.961167 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 04:59:11 dist-test-slave-70st krb5kdc[3909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225951, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 04:59:11 dist-test-slave-70st krb5kdc[3909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225951, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 04:59:11.982933 4420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:11.975294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59326 (local address 127.0.98.1:39765)
0924 04:59:11.975474 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:11.975479 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:11.975490 (+ 11us) server_negotiation.cc:407] Connection header received
0924 04:59:11.975516 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:11.975519 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:11.975532 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:11.975602 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:11.976197 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:11.976915 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:11.977765 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:11.978105 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:11.978175 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:11.980387 (+ 2212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:11.980395 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:11.980395 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:59:11.980403 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:11.981984 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:11.982307 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:11.982309 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:11.982310 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:11.982361 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:11.982486 (+ 125us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:11.982486 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:11.982487 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:11.982606 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:11.982701 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:11.982779 (+ 78us) server_negotiation.cc:299] Negotiation successful
0924 04:59:11.982841 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":118,"threads_started":1}
Sep 24 04:59:11 dist-test-slave-70st krb5kdc[3909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225951, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:59:11.990335 3994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:11.984083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53788 (local address 127.0.98.62:34661)
0924 04:59:11.984122 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:11.984124 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:11.984183 (+ 59us) server_negotiation.cc:407] Connection header received
0924 04:59:11.984233 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:11.984234 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:11.984243 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:11.984293 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:11.984720 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:11.985250 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:11.985940 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:11.986117 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:11.986164 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:11.988185 (+ 2021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:11.988186 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:11.988187 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:59:11.988194 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:11.989475 (+ 1281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:11.989761 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:11.989762 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:11.989762 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:11.989811 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:11.989946 (+ 135us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:11.989946 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:11.989946 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:11.990078 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:11.990142 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:11.990213 (+ 71us) server_negotiation.cc:299] Negotiation successful
0924 04:59:11.990270 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
W20260924 04:59:11.990597 3954 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53788
I20260924 04:59:11.990916 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 4016
I20260924 04:59:11.996013 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 4150
I20260924 04:59:11.999914 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 4284
I20260924 04:59:12.003517 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 3922
2026-09-24T04:59:12Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1689 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225929848825-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:59:12 dist-test-slave-70st krb5kdc[4425](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:59:12 dist-test-slave-70st krb5kdc[4425](info): set up 2 sockets
Sep 24 04:59:12 dist-test-slave-70st krb5kdc[4425](info): commencing operation
krb5kdc: starting...
W20260924 04:59:12.949072 392 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225929848825-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225929848825-392-0/krb5kdc/test-user.keytab.
Sep 24 04:59:13 dist-test-slave-70st krb5kdc[4425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225953, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:59:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:59:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:59:13.067761 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225929848825-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225929848825-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225929848825-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42451
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42451
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:59:13.147284 4438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:59:13.147495 4438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:59:13.147516 4438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:59:13.148591 4438 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:59:13.148623 4438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:59:13.148635 4438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:59:13.148646 4438 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:59:13.148656 4438 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:59:13.150102 4438 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225929848825-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225929848825-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42451
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42451
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4438
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225929848825-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 04:59:13.150291 4438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:59:13.150447 4438 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:59:13.155050 4438 server_base.cc:1034] running on GCE node
W20260924 04:59:13.155058 4443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:59:13.155153 4444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:59:13.155027 4446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:59:13.155483 4438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:59:13.155783 4438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:59:13.156930 4438 hybrid_clock.cc:648] HybridClock initialized: now 1790225953156919 us; error 31 us; skew 500 ppm
Sep 24 04:59:13 dist-test-slave-70st krb5kdc[4425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225953, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:59:13.159443 4438 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:59:13.159808 4438 fs_manager.cc:359] Metadata directory not provided
I20260924 04:59:13.159862 4438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:59:13.159986 4438 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:59:13.161063 4438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225929848825-392-0/minicluster-data/master-0/data/instance:
uuid: "924838fa03364764bee727212a018f85"
format_stamp: "Formatted at 2026-09-24 04:59:13 on dist-test-slave-70st"
server_key: "f90d52d76601e2767a65eee3dfdbe394"
server_key_iv: "a310f8b0dec33e5822208b8b55a34c8c"
server_key_version: "encryptionkey@0"
I20260924 04:59:13.161394 4438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225929848825-392-0/minicluster-data/master-0/wal/instance:
uuid: "924838fa03364764bee727212a018f85"
format_stamp: "Formatted at 2026-09-24 04:59:13 on dist-test-slave-70st"
server_key: "f90d52d76601e2767a65eee3dfdbe394"
server_key_iv: "a310f8b0dec33e5822208b8b55a34c8c"
server_key_version: "encryptionkey@0"
I20260924 04:59:13.162726 4438 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 04:59:13.163466 4452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:13.163679 4438 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 04:59:13.163767 4438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225929848825-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225929848825-392-0/minicluster-data/master-0/wal
uuid: "924838fa03364764bee727212a018f85"
format_stamp: "Formatted at 2026-09-24 04:59:13 on dist-test-slave-70st"
server_key: "f90d52d76601e2767a65eee3dfdbe394"
server_key_iv: "a310f8b0dec33e5822208b8b55a34c8c"
server_key_version: "encryptionkey@0"
I20260924 04:59:13.163834 4438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225929848825-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225929848825-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225929848825-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:59:13.178606 4438 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:59:13.180723 4438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:59:13.180861 4438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:59:13.184639 4438 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42451
I20260924 04:59:13.184670 4504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42451 every 8 connection(s)
I20260924 04:59:13.184958 4438 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:59:13.185868 4505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:59:13.188134 4505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 924838fa03364764bee727212a018f85: Bootstrap starting.
I20260924 04:59:13.188704 4505 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 924838fa03364764bee727212a018f85: Neither blocks nor log segments found. Creating new log.
I20260924 04:59:13.188982 4505 log.cc:824] T 00000000000000000000000000000000 P 924838fa03364764bee727212a018f85: Log is configured to *not* fsync() on all Append() calls
I20260924 04:59:13.189630 4505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 924838fa03364764bee727212a018f85: No bootstrap required, opened a new log
I20260924 04:59:13.191166 4505 raft_consensus.cc:348] T 00000000000000000000000000000000 P 924838fa03364764bee727212a018f85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "924838fa03364764bee727212a018f85" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42451 } }
I20260924 04:59:13.191308 4505 raft_consensus.cc:374] T 00000000000000000000000000000000 P 924838fa03364764bee727212a018f85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:59:13.191330 4505 raft_consensus.cc:729] T 00000000000000000000000000000000 P 924838fa03364764bee727212a018f85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 924838fa03364764bee727212a018f85, State: Initialized, Role: FOLLOWER
I20260924 04:59:13.191461 4505 consensus_queue.cc:260] T 00000000000000000000000000000000 P 924838fa03364764bee727212a018f85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "924838fa03364764bee727212a018f85" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42451 } }
I20260924 04:59:13.191532 4505 raft_consensus.cc:388] T 00000000000000000000000000000000 P 924838fa03364764bee727212a018f85 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:59:13.191555 4505 raft_consensus.cc:482] T 00000000000000000000000000000000 P 924838fa03364764bee727212a018f85 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:59:13.191599 4505 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 924838fa03364764bee727212a018f85 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:59:13.192375 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 4438
I20260924 04:59:13.192484 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225929848825-392-0/minicluster-data/master-0/wal/instance
I20260924 04:59:13.192466 4505 raft_consensus.cc:504] T 00000000000000000000000000000000 P 924838fa03364764bee727212a018f85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "924838fa03364764bee727212a018f85" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42451 } }
I20260924 04:59:13.192569 4505 leader_election.cc:302] T 00000000000000000000000000000000 P 924838fa03364764bee727212a018f85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 924838fa03364764bee727212a018f85; no voters:
I20260924 04:59:13.192678 392 external_mini_cluster.cc:1428] Setting key d32778fd4c2bc85c504fc4c9f5f1c9be
I20260924 04:59:13.192735 4505 leader_election.cc:288] T 00000000000000000000000000000000 P 924838fa03364764bee727212a018f85 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:59:13.192785 4508 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 924838fa03364764bee727212a018f85 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:59:13.192938 4508 raft_consensus.cc:686] T 00000000000000000000000000000000 P 924838fa03364764bee727212a018f85 [term 1 LEADER]: Becoming Leader. State: Replica: 924838fa03364764bee727212a018f85, State: Running, Role: LEADER
I20260924 04:59:13.192997 4505 sys_catalog.cc:564] T 00000000000000000000000000000000 P 924838fa03364764bee727212a018f85 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:59:13.193097 4508 consensus_queue.cc:237] T 00000000000000000000000000000000 P 924838fa03364764bee727212a018f85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "924838fa03364764bee727212a018f85" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42451 } }
I20260924 04:59:13.193490 4508 sys_catalog.cc:455] T 00000000000000000000000000000000 P 924838fa03364764bee727212a018f85 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 924838fa03364764bee727212a018f85. Latest consensus state: current_term: 1 leader_uuid: "924838fa03364764bee727212a018f85" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "924838fa03364764bee727212a018f85" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42451 } } }
I20260924 04:59:13.193497 4510 sys_catalog.cc:455] T 00000000000000000000000000000000 P 924838fa03364764bee727212a018f85 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "924838fa03364764bee727212a018f85" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "924838fa03364764bee727212a018f85" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42451 } } }
I20260924 04:59:13.193650 4508 sys_catalog.cc:458] T 00000000000000000000000000000000 P 924838fa03364764bee727212a018f85 [sys.catalog]: This master's current role is: LEADER
I20260924 04:59:13.193789 4510 sys_catalog.cc:458] T 00000000000000000000000000000000 P 924838fa03364764bee727212a018f85 [sys.catalog]: This master's current role is: LEADER
I20260924 04:59:13.194084 4519 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:59:13.194829 4519 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:59:13.196413 4519 catalog_manager.cc:1347] Generated new cluster ID: ac3cdf9d3d3741b290e3292b6eb84583
I20260924 04:59:13.196463 4519 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 04:59:13 dist-test-slave-70st krb5kdc[4425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225953, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:59:13.204334 4512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:13.193266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54254 (local address 127.0.98.62:42451)
0924 04:59:13.193456 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:13.193512 (+ 56us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:13.193601 (+ 89us) server_negotiation.cc:407] Connection header received
0924 04:59:13.193924 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:13.193933 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:13.194000 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:13.194937 (+ 937us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:13.195054 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:13.196203 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:13.197277 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:13.197902 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:13.197999 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:13.200708 (+ 2709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:13.200713 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:13.200714 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:59:13.200728 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:13.202714 (+ 1986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:13.203097 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:13.203101 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:13.203102 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:13.203219 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:13.203401 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:13.203401 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:13.203402 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:13.203732 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:13.203886 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:13.203955 (+ 69us) server_negotiation.cc:299] Negotiation successful
0924 04:59:13.204109 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":64,"threads_started":1}
I20260924 04:59:13.218708 4519 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:59:13.219493 4519 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:59:13.226584 4519 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 924838fa03364764bee727212a018f85: Generated new TSK 0
I20260924 04:59:13.226800 4519 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225929848825-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225929848825-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:59:13.259853 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225929848825-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225929848825-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225929848825-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225929848825-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42451
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225929848825-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:59:13.339480 4531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:59:13.339705 4531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:59:13.339738 4531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:59:13.340922 4531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:59:13.340963 4531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:59:13.340998 4531 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:59:13.342517 4531 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225929848825-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225929848825-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225929848825-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225929848825-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42451
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4531
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225929848825-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 04:59:13.342762 4531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:59:13.342976 4531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:59:13.347821 4531 server_base.cc:1034] running on GCE node
W20260924 04:59:13.347741 4539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:59:13.347739 4537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:59:13.347741 4536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:59:13.348192 4531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:59:13.348449 4531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:59:13.349596 4531 hybrid_clock.cc:648] HybridClock initialized: now 1790225953349578 us; error 31 us; skew 500 ppm
Sep 24 04:59:13 dist-test-slave-70st krb5kdc[4425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225953, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:59:13.352137 4531 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:59:13.352463 4531 fs_manager.cc:359] Metadata directory not provided
I20260924 04:59:13.352515 4531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:59:13.352639 4531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:59:13.353729 4531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225929848825-392-0/minicluster-data/ts-0/data/instance:
uuid: "075b7df49bf74de8a32a9f3489df31b9"
format_stamp: "Formatted at 2026-09-24 04:59:13 on dist-test-slave-70st"
server_key: "bad0c5e9b17a05d6dcb1ec32c2db56d1"
server_key_iv: "811e728da2252b231d6dd7d0dd7558a5"
server_key_version: "encryptionkey@0"
I20260924 04:59:13.354126 4531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225929848825-392-0/minicluster-data/ts-0/wal/instance:
uuid: "075b7df49bf74de8a32a9f3489df31b9"
format_stamp: "Formatted at 2026-09-24 04:59:13 on dist-test-slave-70st"
server_key: "bad0c5e9b17a05d6dcb1ec32c2db56d1"
server_key_iv: "811e728da2252b231d6dd7d0dd7558a5"
server_key_version: "encryptionkey@0"
I20260924 04:59:13.355378 4531 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 04:59:13.356041 4545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:13.356192 4531 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 04:59:13.356276 4531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225929848825-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225929848825-392-0/minicluster-data/ts-0/wal
uuid: "075b7df49bf74de8a32a9f3489df31b9"
format_stamp: "Formatted at 2026-09-24 04:59:13 on dist-test-slave-70st"
server_key: "bad0c5e9b17a05d6dcb1ec32c2db56d1"
server_key_iv: "811e728da2252b231d6dd7d0dd7558a5"
server_key_version: "encryptionkey@0"
I20260924 04:59:13.356333 4531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225929848825-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225929848825-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225929848825-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:59:13.379745 4531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:59:13.381892 4531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:59:13.382041 4531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:59:13.382261 4531 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:59:13.382567 4531 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:59:13.382601 4531 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:13.382628 4531 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:59:13.382653 4531 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:13.387997 4531 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45753
I20260924 04:59:13.388034 4658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45753 every 8 connection(s)
I20260924 04:59:13.388286 4531 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225929848825-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 04:59:13 dist-test-slave-70st krb5kdc[4425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225953, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:59:13.394979 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 4531
I20260924 04:59:13.395071 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225929848825-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:59:13.395257 392 external_mini_cluster.cc:1428] Setting key 90faefc39b502ffcf69bc618e8f17cfb
I20260924 04:59:13.397603 4512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:13.388943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47719 (local address 127.0.98.62:42451)
0924 04:59:13.388987 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:13.388989 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:13.389212 (+ 223us) server_negotiation.cc:407] Connection header received
0924 04:59:13.389739 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:13.389741 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:13.389756 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:13.389803 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:13.390724 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:13.391224 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:13.392183 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:13.392377 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:13.392432 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:13.394612 (+ 2180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:13.394614 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:13.394614 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:59:13.394621 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:13.396175 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:13.396611 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:13.396612 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:13.396612 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:13.396669 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:13.396896 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:13.396897 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:13.396897 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:13.397019 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:13.397091 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:13.397388 (+ 297us) server_negotiation.cc:299] Negotiation successful
0924 04:59:13.397503 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 04:59:13.397680 4661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:13.388940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42451 (local address 127.0.98.1:47719)
0924 04:59:13.389090 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:13.389126 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:13.389647 (+ 521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:13.389883 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:13.389887 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:13.390145 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:13.390629 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:13.390634 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:13.391302 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:13.391303 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:13.392101 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:13.392103 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:13.392449 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:13.392450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:13.392563 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:13.393035 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:13.393043 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:13.394509 (+ 1466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:59:13.396254 (+ 1745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:13.396261 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:13.396263 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:13.396534 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:13.396721 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:13.396722 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:13.396723 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:13.396843 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:13.397086 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:59:13.397092 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:59:13.397260 (+ 168us) client_negotiation.cc:769] Sending connection context
0924 04:59:13.397346 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 04:59:13.397509 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":70,"threads_started":1}
I20260924 04:59:13.398289 4659 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42451
I20260924 04:59:13.398386 4659 heartbeater.cc:464] Registering TS with master...
I20260924 04:59:13.398537 4659 heartbeater.cc:511] Master 127.0.98.62:42451 requested a full tablet report, sending...
I20260924 04:59:13.399080 4469 ts_manager.cc:194] Registered new tserver with Master: 075b7df49bf74de8a32a9f3489df31b9 (127.0.98.1:45753)
I20260924 04:59:13.399971 4469 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:47719
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225929848825-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225929848825-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:59:13.420208 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225929848825-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225929848825-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225929848825-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225929848825-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42451
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225929848825-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:59:13.499207 4664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:59:13.499423 4664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:59:13.499459 4664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:59:13.500629 4664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:59:13.500671 4664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:59:13.500706 4664 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:59:13.502270 4664 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225929848825-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225929848825-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225929848825-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225929848825-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42451
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4664
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225929848825-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 04:59:13.502518 4664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:59:13.502722 4664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:59:13.507603 4669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:59:13.507624 4672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:59:13.507663 4664 server_base.cc:1034] running on GCE node
W20260924 04:59:13.507577 4670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:59:13.508010 4664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:59:13.508227 4664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:59:13.509394 4664 hybrid_clock.cc:648] HybridClock initialized: now 1790225953509375 us; error 37 us; skew 500 ppm
Sep 24 04:59:13 dist-test-slave-70st krb5kdc[4425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225953, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:59:13.511961 4664 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:59:13.512293 4664 fs_manager.cc:359] Metadata directory not provided
I20260924 04:59:13.512341 4664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:59:13.512456 4664 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:59:13.513576 4664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225929848825-392-0/minicluster-data/ts-1/data/instance:
uuid: "70c4cdf67d7b425197eb5609a8ee0f04"
format_stamp: "Formatted at 2026-09-24 04:59:13 on dist-test-slave-70st"
server_key: "cbfb77ebb7ddd4a643987909c8af6735"
server_key_iv: "74f633cf24f1da9b68e93bfc76191e0b"
server_key_version: "encryptionkey@0"
I20260924 04:59:13.513906 4664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225929848825-392-0/minicluster-data/ts-1/wal/instance:
uuid: "70c4cdf67d7b425197eb5609a8ee0f04"
format_stamp: "Formatted at 2026-09-24 04:59:13 on dist-test-slave-70st"
server_key: "cbfb77ebb7ddd4a643987909c8af6735"
server_key_iv: "74f633cf24f1da9b68e93bfc76191e0b"
server_key_version: "encryptionkey@0"
I20260924 04:59:13.515260 4664 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 04:59:13.516003 4678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:13.516158 4664 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 04:59:13.516238 4664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225929848825-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225929848825-392-0/minicluster-data/ts-1/wal
uuid: "70c4cdf67d7b425197eb5609a8ee0f04"
format_stamp: "Formatted at 2026-09-24 04:59:13 on dist-test-slave-70st"
server_key: "cbfb77ebb7ddd4a643987909c8af6735"
server_key_iv: "74f633cf24f1da9b68e93bfc76191e0b"
server_key_version: "encryptionkey@0"
I20260924 04:59:13.516295 4664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225929848825-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225929848825-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225929848825-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:59:13.538787 4664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:59:13.540620 4664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:59:13.540755 4664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:59:13.540967 4664 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:59:13.541270 4664 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:59:13.541303 4664 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:13.541332 4664 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:59:13.541352 4664 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:13.547142 4664 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44441
I20260924 04:59:13.547184 4791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44441 every 8 connection(s)
I20260924 04:59:13.547446 4664 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225929848825-392-0/minicluster-data/ts-1/data/info.pb
Sep 24 04:59:13 dist-test-slave-70st krb5kdc[4425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225953, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:59:13.555018 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 4664
I20260924 04:59:13.555125 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225929848825-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:59:13.555302 392 external_mini_cluster.cc:1428] Setting key e1d15dc19df7fe8c69b25323e2854d1f
I20260924 04:59:13.556173 4512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:13.548126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57723 (local address 127.0.98.62:42451)
0924 04:59:13.548178 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:13.548180 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:13.548355 (+ 175us) server_negotiation.cc:407] Connection header received
0924 04:59:13.548828 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:13.548829 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:13.548839 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:13.548885 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:13.549829 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:13.550340 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:13.551289 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:13.551466 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:13.551510 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:13.553592 (+ 2082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:13.553594 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:13.553594 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:59:13.553602 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:13.554704 (+ 1102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:13.555202 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:13.555204 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:13.555204 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:13.555264 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:13.555538 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:13.555539 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:13.555539 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:13.555680 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:13.555746 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:13.556031 (+ 285us) server_negotiation.cc:299] Negotiation successful
0924 04:59:13.556109 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 04:59:13.556430 4794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:13.548100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42451 (local address 127.0.98.2:57723)
0924 04:59:13.548250 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:13.548287 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:13.548750 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:13.548962 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:13.548965 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:13.549216 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:13.549761 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:13.549768 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:13.550405 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:13.550406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:13.551229 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:13.551231 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:13.551521 (+ 290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:13.551522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:13.551618 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:13.552107 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:13.552116 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:13.553494 (+ 1378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:59:13.554778 (+ 1284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:13.554785 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:13.554786 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:13.555116 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:13.555323 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:13.555324 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:13.555324 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:13.555476 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:13.555741 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:59:13.555746 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:59:13.555886 (+ 140us) client_negotiation.cc:769] Sending connection context
0924 04:59:13.555999 (+ 113us) client_negotiation.cc:240] Negotiation successful
0924 04:59:13.556179 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":86,"threads_started":1}
I20260924 04:59:13.557598 4792 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42451
I20260924 04:59:13.557714 4792 heartbeater.cc:464] Registering TS with master...
I20260924 04:59:13.557876 4792 heartbeater.cc:511] Master 127.0.98.62:42451 requested a full tablet report, sending...
I20260924 04:59:13.558226 4469 ts_manager.cc:194] Registered new tserver with Master: 70c4cdf67d7b425197eb5609a8ee0f04 (127.0.98.2:44441)
I20260924 04:59:13.558612 4469 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:57723
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225929848825-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225929848825-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:59:13.580092 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225929848825-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225929848825-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225929848825-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225929848825-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42451
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225929848825-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:59:13.658835 4797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:59:13.659057 4797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:59:13.659083 4797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:59:13.660251 4797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:59:13.660292 4797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:59:13.660318 4797 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:59:13.661870 4797 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225929848825-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225929848825-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225929848825-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225929848825-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42451
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4797
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225929848825-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 04:59:13.662096 4797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:59:13.662322 4797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:59:13.667124 4802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:59:13.667107 4803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:59:13.667097 4805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:59:13.667363 4797 server_base.cc:1034] running on GCE node
I20260924 04:59:13.667579 4797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:59:13.667800 4797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:59:13.668934 4797 hybrid_clock.cc:648] HybridClock initialized: now 1790225953668918 us; error 35 us; skew 500 ppm
Sep 24 04:59:13 dist-test-slave-70st krb5kdc[4425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225953, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:59:13.671325 4797 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:59:13.671651 4797 fs_manager.cc:359] Metadata directory not provided
I20260924 04:59:13.671700 4797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:59:13.671802 4797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:59:13.672832 4797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225929848825-392-0/minicluster-data/ts-2/data/instance:
uuid: "fbf5946ab1114e85bd81469554f8c893"
format_stamp: "Formatted at 2026-09-24 04:59:13 on dist-test-slave-70st"
server_key: "0123935f31917a3962f928651d650013"
server_key_iv: "9c989748e2fad64f24a5dd7be310f1e6"
server_key_version: "encryptionkey@0"
I20260924 04:59:13.673159 4797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225929848825-392-0/minicluster-data/ts-2/wal/instance:
uuid: "fbf5946ab1114e85bd81469554f8c893"
format_stamp: "Formatted at 2026-09-24 04:59:13 on dist-test-slave-70st"
server_key: "0123935f31917a3962f928651d650013"
server_key_iv: "9c989748e2fad64f24a5dd7be310f1e6"
server_key_version: "encryptionkey@0"
I20260924 04:59:13.674479 4797 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 04:59:13.675117 4811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:13.675256 4797 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 04:59:13.675330 4797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225929848825-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225929848825-392-0/minicluster-data/ts-2/wal
uuid: "fbf5946ab1114e85bd81469554f8c893"
format_stamp: "Formatted at 2026-09-24 04:59:13 on dist-test-slave-70st"
server_key: "0123935f31917a3962f928651d650013"
server_key_iv: "9c989748e2fad64f24a5dd7be310f1e6"
server_key_version: "encryptionkey@0"
I20260924 04:59:13.675396 4797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225929848825-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225929848825-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225929848825-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:59:13.700904 4797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:59:13.702703 4797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:59:13.702823 4797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:59:13.703039 4797 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:59:13.703337 4797 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:59:13.703372 4797 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:13.703399 4797 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:59:13.703423 4797 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:13.708755 4797 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40661
I20260924 04:59:13.708796 4924 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40661 every 8 connection(s)
I20260924 04:59:13.709052 4797 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225929848825-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 04:59:13 dist-test-slave-70st krb5kdc[4425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225953, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:59:13.714931 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 4797
I20260924 04:59:13.715029 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225929848825-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:59:13.715266 392 external_mini_cluster.cc:1428] Setting key 2b09b9751bbb501348d3024f374f2a39
I20260924 04:59:13.717927 4512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:13.709720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45235 (local address 127.0.98.62:42451)
0924 04:59:13.709793 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:13.709795 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:13.709966 (+ 171us) server_negotiation.cc:407] Connection header received
0924 04:59:13.710421 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:13.710444 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:13.710454 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:13.710495 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:13.711407 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:13.711904 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:13.712846 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:13.713028 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:13.713072 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:13.715335 (+ 2263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:13.715337 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:13.715337 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:59:13.715346 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:13.716606 (+ 1260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:13.716988 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:13.716989 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:13.716989 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:13.717051 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:13.717285 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:13.717285 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:13.717286 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:13.717397 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:13.717487 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:13.717750 (+ 263us) server_negotiation.cc:299] Negotiation successful
0924 04:59:13.717837 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 04:59:13.718098 4927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:13.709692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42451 (local address 127.0.98.3:45235)
0924 04:59:13.709842 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:13.709877 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:13.710334 (+ 457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:13.710564 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:13.710568 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:13.710839 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:13.711320 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:13.711325 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:13.711975 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:13.711975 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:13.712784 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:13.712786 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:13.713089 (+ 303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:13.713089 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:13.713193 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:13.713704 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:13.713712 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:13.715236 (+ 1524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:59:13.716692 (+ 1456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:13.716699 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:13.716701 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:13.716920 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:13.717105 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:13.717106 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:13.717106 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:13.717218 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:13.717476 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:59:13.717483 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:59:13.717628 (+ 145us) client_negotiation.cc:769] Sending connection context
0924 04:59:13.717721 (+ 93us) client_negotiation.cc:240] Negotiation successful
0924 04:59:13.717880 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":81,"threads_started":1}
I20260924 04:59:13.718489 4925 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42451
I20260924 04:59:13.718585 4925 heartbeater.cc:464] Registering TS with master...
I20260924 04:59:13.718732 4925 heartbeater.cc:511] Master 127.0.98.62:42451 requested a full tablet report, sending...
I20260924 04:59:13.719072 4469 ts_manager.cc:194] Registered new tserver with Master: fbf5946ab1114e85bd81469554f8c893 (127.0.98.3:40661)
I20260924 04:59:13.719453 4469 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45235
I20260924 04:59:13.726977 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:59:13.733904 4512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:13.728297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54260 (local address 127.0.98.62:42451)
0924 04:59:13.728379 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:13.728382 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:13.728456 (+ 74us) server_negotiation.cc:407] Connection header received
0924 04:59:13.728515 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:13.728516 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:13.728527 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:13.728574 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:13.729123 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:13.729661 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:13.730342 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:13.730522 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:13.730577 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:13.731723 (+ 1146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:13.731724 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:13.731725 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:59:13.731733 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:13.732848 (+ 1115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:13.733147 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:13.733148 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:13.733148 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:13.733198 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:13.733348 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:13.733349 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:13.733349 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:13.733533 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:13.733619 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:13.733737 (+ 118us) server_negotiation.cc:299] Negotiation successful
0924 04:59:13.733813 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 04:59:13.735505 4469 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54260:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:59:13.736977 4469 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:59:13.743331 4938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:13.739155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45753 (local address 127.0.0.1:36034)
0924 04:59:13.739313 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:13.739350 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:13.739417 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:13.739766 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:13.739769 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:13.739772 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:59:13.739843 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:13.739845 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:13.740932 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:13.740933 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:13.742090 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:13.742093 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:13.742899 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:13.742900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:13.743072 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:13.743093 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 04:59:13.743183 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 04:59:13.743246 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":49,"threads_started":1}
I20260924 04:59:13.743345 4939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:13.739429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36034 (local address 127.0.98.1:45753)
0924 04:59:13.739592 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:13.739597 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:13.739608 (+ 11us) server_negotiation.cc:407] Connection header received
0924 04:59:13.739649 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:13.739651 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:13.739668 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:13.739757 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:59:13.739991 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:13.740833 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:13.742179 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:13.742830 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:13.742891 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:13.742954 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:13.743212 (+ 258us) server_negotiation.cc:299] Negotiation successful
0924 04:59:13.743258 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":83,"threads_started":1}
I20260924 04:59:13.743567 4937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:13.739161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44441 (local address 127.0.0.1:54628)
0924 04:59:13.739512 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:13.739520 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:13.739554 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:13.740022 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:13.740024 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:13.740025 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:59:13.740062 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:13.740063 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:13.741407 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:13.741408 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:13.742375 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:13.742377 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:13.743338 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:13.743338 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:13.743390 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:13.743393 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 04:59:13.743436 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 04:59:13.743462 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":96,"threads_started":1}
I20260924 04:59:13.743570 4936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:13.739161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40661 (local address 127.0.0.1:33004)
0924 04:59:13.739581 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:13.739590 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:13.739626 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:13.739895 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:13.739896 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:13.739899 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:59:13.739945 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:13.739947 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:13.741406 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:13.741407 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:13.742471 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:13.742473 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:13.743292 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:13.743293 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:13.743385 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:13.743389 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 04:59:13.743419 (+ 30us) client_negotiation.cc:240] Negotiation successful
0924 04:59:13.743462 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"mutex_wait_us":2,"thread_start_us":98,"threads_started":1}
I20260924 04:59:13.743726 4941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:13.739608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33004 (local address 127.0.98.3:40661)
0924 04:59:13.739751 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:13.739756 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:13.739765 (+ 9us) server_negotiation.cc:407] Connection header received
0924 04:59:13.739798 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:13.739800 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:13.739813 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:13.739892 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:59:13.740034 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:13.741251 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:13.742545 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:13.743206 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:13.743509 (+ 303us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:13.743578 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:13.743603 (+ 25us) server_negotiation.cc:299] Negotiation successful
0924 04:59:13.743647 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":79,"threads_started":1}
I20260924 04:59:13.743865 4940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:13.739419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54628 (local address 127.0.98.2:44441)
0924 04:59:13.739806 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:13.739810 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:13.739819 (+ 9us) server_negotiation.cc:407] Connection header received
0924 04:59:13.739854 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:13.739856 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:13.739870 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:13.739941 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:59:13.740130 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:13.741300 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:13.742451 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:13.743178 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:13.743249 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:13.743309 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:13.743729 (+ 420us) server_negotiation.cc:299] Negotiation successful
0924 04:59:13.743772 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":87,"threads_started":1}
I20260924 04:59:13.744473 4859 tablet_service.cc:1467] Processing CreateTablet for tablet 5fc55278c4f64ae2ae3796e935f383e9 (DEFAULT_TABLE table=test-table [id=7231ee52c2a74f20896da646a350a071]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:59:13.744568 4726 tablet_service.cc:1467] Processing CreateTablet for tablet 5fc55278c4f64ae2ae3796e935f383e9 (DEFAULT_TABLE table=test-table [id=7231ee52c2a74f20896da646a350a071]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:59:13.744837 4859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fc55278c4f64ae2ae3796e935f383e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:59:13.744849 4726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fc55278c4f64ae2ae3796e935f383e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:59:13.747753 4943 tablet_bootstrap.cc:492] T 5fc55278c4f64ae2ae3796e935f383e9 P 70c4cdf67d7b425197eb5609a8ee0f04: Bootstrap starting.
I20260924 04:59:13.747802 4593 tablet_service.cc:1467] Processing CreateTablet for tablet 5fc55278c4f64ae2ae3796e935f383e9 (DEFAULT_TABLE table=test-table [id=7231ee52c2a74f20896da646a350a071]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:59:13.748246 4593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fc55278c4f64ae2ae3796e935f383e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:59:13.748467 4943 tablet_bootstrap.cc:654] T 5fc55278c4f64ae2ae3796e935f383e9 P 70c4cdf67d7b425197eb5609a8ee0f04: Neither blocks nor log segments found. Creating new log.
I20260924 04:59:13.748951 4943 log.cc:824] T 5fc55278c4f64ae2ae3796e935f383e9 P 70c4cdf67d7b425197eb5609a8ee0f04: Log is configured to *not* fsync() on all Append() calls
I20260924 04:59:13.749294 4942 tablet_bootstrap.cc:492] T 5fc55278c4f64ae2ae3796e935f383e9 P fbf5946ab1114e85bd81469554f8c893: Bootstrap starting.
I20260924 04:59:13.749723 4943 tablet_bootstrap.cc:492] T 5fc55278c4f64ae2ae3796e935f383e9 P 70c4cdf67d7b425197eb5609a8ee0f04: No bootstrap required, opened a new log
I20260924 04:59:13.749794 4943 ts_tablet_manager.cc:1397] T 5fc55278c4f64ae2ae3796e935f383e9 P 70c4cdf67d7b425197eb5609a8ee0f04: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 04:59:13.749974 4942 tablet_bootstrap.cc:654] T 5fc55278c4f64ae2ae3796e935f383e9 P fbf5946ab1114e85bd81469554f8c893: Neither blocks nor log segments found. Creating new log.
I20260924 04:59:13.750509 4942 log.cc:824] T 5fc55278c4f64ae2ae3796e935f383e9 P fbf5946ab1114e85bd81469554f8c893: Log is configured to *not* fsync() on all Append() calls
I20260924 04:59:13.751312 4942 tablet_bootstrap.cc:492] T 5fc55278c4f64ae2ae3796e935f383e9 P fbf5946ab1114e85bd81469554f8c893: No bootstrap required, opened a new log
I20260924 04:59:13.751386 4942 ts_tablet_manager.cc:1397] T 5fc55278c4f64ae2ae3796e935f383e9 P fbf5946ab1114e85bd81469554f8c893: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 04:59:13.751996 4943 raft_consensus.cc:348] T 5fc55278c4f64ae2ae3796e935f383e9 P 70c4cdf67d7b425197eb5609a8ee0f04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075b7df49bf74de8a32a9f3489df31b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45753 } } peers { permanent_uuid: "fbf5946ab1114e85bd81469554f8c893" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40661 } } peers { permanent_uuid: "70c4cdf67d7b425197eb5609a8ee0f04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44441 } }
I20260924 04:59:13.752178 4943 raft_consensus.cc:374] T 5fc55278c4f64ae2ae3796e935f383e9 P 70c4cdf67d7b425197eb5609a8ee0f04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:59:13.752213 4943 raft_consensus.cc:729] T 5fc55278c4f64ae2ae3796e935f383e9 P 70c4cdf67d7b425197eb5609a8ee0f04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70c4cdf67d7b425197eb5609a8ee0f04, State: Initialized, Role: FOLLOWER
I20260924 04:59:13.752351 4943 consensus_queue.cc:260] T 5fc55278c4f64ae2ae3796e935f383e9 P 70c4cdf67d7b425197eb5609a8ee0f04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075b7df49bf74de8a32a9f3489df31b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45753 } } peers { permanent_uuid: "fbf5946ab1114e85bd81469554f8c893" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40661 } } peers { permanent_uuid: "70c4cdf67d7b425197eb5609a8ee0f04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44441 } }
I20260924 04:59:13.752626 4943 ts_tablet_manager.cc:1428] T 5fc55278c4f64ae2ae3796e935f383e9 P 70c4cdf67d7b425197eb5609a8ee0f04: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260924 04:59:13.752646 4792 heartbeater.cc:503] Master 127.0.98.62:42451 was elected leader, sending a full tablet report...
I20260924 04:59:13.753500 4946 tablet_bootstrap.cc:492] T 5fc55278c4f64ae2ae3796e935f383e9 P 075b7df49bf74de8a32a9f3489df31b9: Bootstrap starting.
I20260924 04:59:13.753845 4942 raft_consensus.cc:348] T 5fc55278c4f64ae2ae3796e935f383e9 P fbf5946ab1114e85bd81469554f8c893 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075b7df49bf74de8a32a9f3489df31b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45753 } } peers { permanent_uuid: "fbf5946ab1114e85bd81469554f8c893" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40661 } } peers { permanent_uuid: "70c4cdf67d7b425197eb5609a8ee0f04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44441 } }
I20260924 04:59:13.754014 4942 raft_consensus.cc:374] T 5fc55278c4f64ae2ae3796e935f383e9 P fbf5946ab1114e85bd81469554f8c893 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:59:13.754055 4942 raft_consensus.cc:729] T 5fc55278c4f64ae2ae3796e935f383e9 P fbf5946ab1114e85bd81469554f8c893 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbf5946ab1114e85bd81469554f8c893, State: Initialized, Role: FOLLOWER
I20260924 04:59:13.754197 4946 tablet_bootstrap.cc:654] T 5fc55278c4f64ae2ae3796e935f383e9 P 075b7df49bf74de8a32a9f3489df31b9: Neither blocks nor log segments found. Creating new log.
I20260924 04:59:13.754166 4942 consensus_queue.cc:260] T 5fc55278c4f64ae2ae3796e935f383e9 P fbf5946ab1114e85bd81469554f8c893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075b7df49bf74de8a32a9f3489df31b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45753 } } peers { permanent_uuid: "fbf5946ab1114e85bd81469554f8c893" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40661 } } peers { permanent_uuid: "70c4cdf67d7b425197eb5609a8ee0f04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44441 } }
I20260924 04:59:13.754493 4942 ts_tablet_manager.cc:1428] T 5fc55278c4f64ae2ae3796e935f383e9 P fbf5946ab1114e85bd81469554f8c893: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 04:59:13.754506 4925 heartbeater.cc:503] Master 127.0.98.62:42451 was elected leader, sending a full tablet report...
I20260924 04:59:13.754767 4946 log.cc:824] T 5fc55278c4f64ae2ae3796e935f383e9 P 075b7df49bf74de8a32a9f3489df31b9: Log is configured to *not* fsync() on all Append() calls
I20260924 04:59:13.755554 4946 tablet_bootstrap.cc:492] T 5fc55278c4f64ae2ae3796e935f383e9 P 075b7df49bf74de8a32a9f3489df31b9: No bootstrap required, opened a new log
I20260924 04:59:13.755610 4946 ts_tablet_manager.cc:1397] T 5fc55278c4f64ae2ae3796e935f383e9 P 075b7df49bf74de8a32a9f3489df31b9: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260924 04:59:13.756961 4946 raft_consensus.cc:348] T 5fc55278c4f64ae2ae3796e935f383e9 P 075b7df49bf74de8a32a9f3489df31b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075b7df49bf74de8a32a9f3489df31b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45753 } } peers { permanent_uuid: "fbf5946ab1114e85bd81469554f8c893" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40661 } } peers { permanent_uuid: "70c4cdf67d7b425197eb5609a8ee0f04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44441 } }
I20260924 04:59:13.757084 4946 raft_consensus.cc:374] T 5fc55278c4f64ae2ae3796e935f383e9 P 075b7df49bf74de8a32a9f3489df31b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:59:13.757104 4946 raft_consensus.cc:729] T 5fc55278c4f64ae2ae3796e935f383e9 P 075b7df49bf74de8a32a9f3489df31b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 075b7df49bf74de8a32a9f3489df31b9, State: Initialized, Role: FOLLOWER
I20260924 04:59:13.757216 4946 consensus_queue.cc:260] T 5fc55278c4f64ae2ae3796e935f383e9 P 075b7df49bf74de8a32a9f3489df31b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075b7df49bf74de8a32a9f3489df31b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45753 } } peers { permanent_uuid: "fbf5946ab1114e85bd81469554f8c893" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40661 } } peers { permanent_uuid: "70c4cdf67d7b425197eb5609a8ee0f04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44441 } }
I20260924 04:59:13.757503 4946 ts_tablet_manager.cc:1428] T 5fc55278c4f64ae2ae3796e935f383e9 P 075b7df49bf74de8a32a9f3489df31b9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 04:59:13.757500 4659 heartbeater.cc:503] Master 127.0.98.62:42451 was elected leader, sending a full tablet report...
W20260924 04:59:13.797922 4793 tablet.cc:2369] T 5fc55278c4f64ae2ae3796e935f383e9 P 70c4cdf67d7b425197eb5609a8ee0f04: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 04:59:13.888970 4660 tablet.cc:2369] T 5fc55278c4f64ae2ae3796e935f383e9 P 075b7df49bf74de8a32a9f3489df31b9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:59:13.924031 4950 raft_consensus.cc:482] T 5fc55278c4f64ae2ae3796e935f383e9 P 075b7df49bf74de8a32a9f3489df31b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:59:13.924170 4950 raft_consensus.cc:504] T 5fc55278c4f64ae2ae3796e935f383e9 P 075b7df49bf74de8a32a9f3489df31b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075b7df49bf74de8a32a9f3489df31b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45753 } } peers { permanent_uuid: "fbf5946ab1114e85bd81469554f8c893" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40661 } } peers { permanent_uuid: "70c4cdf67d7b425197eb5609a8ee0f04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44441 } }
I20260924 04:59:13.924479 4950 leader_election.cc:288] T 5fc55278c4f64ae2ae3796e935f383e9 P 075b7df49bf74de8a32a9f3489df31b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fbf5946ab1114e85bd81469554f8c893 (127.0.98.3:40661), 70c4cdf67d7b425197eb5609a8ee0f04 (127.0.98.2:44441)
I20260924 04:59:13.927371 4941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:13.924737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44825 (local address 127.0.98.3:40661)
0924 04:59:13.924781 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:13.924783 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:13.924890 (+ 107us) server_negotiation.cc:407] Connection header received
0924 04:59:13.924939 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:13.924940 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:13.924952 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:13.925001 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:59:13.925135 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:13.925662 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:13.926694 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:13.927067 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:13.927126 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:13.927138 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:13.927272 (+ 134us) server_negotiation.cc:299] Negotiation successful
0924 04:59:13.927309 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 04:59:13.927413 4940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:13.924704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41797 (local address 127.0.98.2:44441)
0924 04:59:13.924751 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:13.924753 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:13.924885 (+ 132us) server_negotiation.cc:407] Connection header received
0924 04:59:13.924940 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:13.924941 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:13.924955 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:13.924999 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:59:13.925146 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:13.925626 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:13.926697 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:13.927097 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:13.927157 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:13.927167 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:13.927294 (+ 127us) server_negotiation.cc:299] Negotiation successful
0924 04:59:13.927345 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 04:59:13.927435 4952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:13.924652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44441 (local address 127.0.98.1:41797)
0924 04:59:13.924808 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:13.924819 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:13.924879 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:13.925011 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:13.925012 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:13.925014 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:59:13.925090 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:13.925092 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:13.925700 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:13.925701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:13.926614 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:13.926617 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:13.927166 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:13.927167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:13.927236 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:13.927239 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 04:59:13.927283 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 04:59:13.927324 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"mutex_wait_us":58,"thread_start_us":90,"threads_started":1}
I20260924 04:59:13.927608 4951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:13.924652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40661 (local address 127.0.98.1:44825)
0924 04:59:13.924821 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:13.924830 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:13.924887 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:13.925010 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:13.925011 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:13.925012 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:59:13.925083 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:13.925085 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:13.925735 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:13.925736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:13.926611 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:13.926613 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:13.927138 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:13.927140 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:13.927200 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:13.927204 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 04:59:13.927247 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 04:59:13.927292 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":5,"thread_start_us":83,"threads_started":1}
I20260924 04:59:13.927785 4746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fc55278c4f64ae2ae3796e935f383e9" candidate_uuid: "075b7df49bf74de8a32a9f3489df31b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70c4cdf67d7b425197eb5609a8ee0f04" is_pre_election: true
I20260924 04:59:13.927888 4879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fc55278c4f64ae2ae3796e935f383e9" candidate_uuid: "075b7df49bf74de8a32a9f3489df31b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbf5946ab1114e85bd81469554f8c893" is_pre_election: true
I20260924 04:59:13.927958 4746 raft_consensus.cc:2454] T 5fc55278c4f64ae2ae3796e935f383e9 P 70c4cdf67d7b425197eb5609a8ee0f04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075b7df49bf74de8a32a9f3489df31b9 in term 0.
I20260924 04:59:13.928041 4879 raft_consensus.cc:2454] T 5fc55278c4f64ae2ae3796e935f383e9 P fbf5946ab1114e85bd81469554f8c893 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075b7df49bf74de8a32a9f3489df31b9 in term 0.
I20260924 04:59:13.928158 4547 leader_election.cc:302] T 5fc55278c4f64ae2ae3796e935f383e9 P 075b7df49bf74de8a32a9f3489df31b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075b7df49bf74de8a32a9f3489df31b9, 70c4cdf67d7b425197eb5609a8ee0f04; no voters:
I20260924 04:59:13.928287 4950 raft_consensus.cc:2781] T 5fc55278c4f64ae2ae3796e935f383e9 P 075b7df49bf74de8a32a9f3489df31b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:59:13.928341 4950 raft_consensus.cc:482] T 5fc55278c4f64ae2ae3796e935f383e9 P 075b7df49bf74de8a32a9f3489df31b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:59:13.928359 4950 raft_consensus.cc:3037] T 5fc55278c4f64ae2ae3796e935f383e9 P 075b7df49bf74de8a32a9f3489df31b9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:59:13.929028 4950 raft_consensus.cc:504] T 5fc55278c4f64ae2ae3796e935f383e9 P 075b7df49bf74de8a32a9f3489df31b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075b7df49bf74de8a32a9f3489df31b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45753 } } peers { permanent_uuid: "fbf5946ab1114e85bd81469554f8c893" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40661 } } peers { permanent_uuid: "70c4cdf67d7b425197eb5609a8ee0f04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44441 } }
I20260924 04:59:13.929211 4950 leader_election.cc:288] T 5fc55278c4f64ae2ae3796e935f383e9 P 075b7df49bf74de8a32a9f3489df31b9 [CANDIDATE]: Term 1 election: Requested vote from peers fbf5946ab1114e85bd81469554f8c893 (127.0.98.3:40661), 70c4cdf67d7b425197eb5609a8ee0f04 (127.0.98.2:44441)
I20260924 04:59:13.929293 4879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fc55278c4f64ae2ae3796e935f383e9" candidate_uuid: "075b7df49bf74de8a32a9f3489df31b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbf5946ab1114e85bd81469554f8c893"
I20260924 04:59:13.929324 4746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fc55278c4f64ae2ae3796e935f383e9" candidate_uuid: "075b7df49bf74de8a32a9f3489df31b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70c4cdf67d7b425197eb5609a8ee0f04"
I20260924 04:59:13.929360 4879 raft_consensus.cc:3037] T 5fc55278c4f64ae2ae3796e935f383e9 P fbf5946ab1114e85bd81469554f8c893 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:59:13.929385 4746 raft_consensus.cc:3037] T 5fc55278c4f64ae2ae3796e935f383e9 P 70c4cdf67d7b425197eb5609a8ee0f04 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:59:13.930066 4746 raft_consensus.cc:2454] T 5fc55278c4f64ae2ae3796e935f383e9 P 70c4cdf67d7b425197eb5609a8ee0f04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075b7df49bf74de8a32a9f3489df31b9 in term 1.
I20260924 04:59:13.930122 4879 raft_consensus.cc:2454] T 5fc55278c4f64ae2ae3796e935f383e9 P fbf5946ab1114e85bd81469554f8c893 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075b7df49bf74de8a32a9f3489df31b9 in term 1.
I20260924 04:59:13.930275 4547 leader_election.cc:302] T 5fc55278c4f64ae2ae3796e935f383e9 P 075b7df49bf74de8a32a9f3489df31b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075b7df49bf74de8a32a9f3489df31b9, 70c4cdf67d7b425197eb5609a8ee0f04; no voters:
I20260924 04:59:13.930379 4950 raft_consensus.cc:2781] T 5fc55278c4f64ae2ae3796e935f383e9 P 075b7df49bf74de8a32a9f3489df31b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:59:13.930523 4950 raft_consensus.cc:686] T 5fc55278c4f64ae2ae3796e935f383e9 P 075b7df49bf74de8a32a9f3489df31b9 [term 1 LEADER]: Becoming Leader. State: Replica: 075b7df49bf74de8a32a9f3489df31b9, State: Running, Role: LEADER
I20260924 04:59:13.930604 4950 consensus_queue.cc:237] T 5fc55278c4f64ae2ae3796e935f383e9 P 075b7df49bf74de8a32a9f3489df31b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075b7df49bf74de8a32a9f3489df31b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45753 } } peers { permanent_uuid: "fbf5946ab1114e85bd81469554f8c893" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40661 } } peers { permanent_uuid: "70c4cdf67d7b425197eb5609a8ee0f04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44441 } }
I20260924 04:59:13.931399 4469 catalog_manager.cc:5495] T 5fc55278c4f64ae2ae3796e935f383e9 P 075b7df49bf74de8a32a9f3489df31b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 075b7df49bf74de8a32a9f3489df31b9 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "075b7df49bf74de8a32a9f3489df31b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075b7df49bf74de8a32a9f3489df31b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45753 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fbf5946ab1114e85bd81469554f8c893" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "70c4cdf67d7b425197eb5609a8ee0f04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44441 } health_report { overall_health: UNKNOWN } } }
W20260924 04:59:13.959538 4926 tablet.cc:2369] T 5fc55278c4f64ae2ae3796e935f383e9 P fbf5946ab1114e85bd81469554f8c893: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:59:13.971740 4939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:13.969558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36046 (local address 127.0.98.1:45753)
0924 04:59:13.969620 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:13.969622 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:13.969631 (+ 9us) server_negotiation.cc:407] Connection header received
0924 04:59:13.969669 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:13.969670 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:13.969689 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:13.969748 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:59:13.969866 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:13.970344 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:13.971060 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:13.971238 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:13.971291 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:13.971434 (+ 143us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:59:13.971543 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:59:13.971620 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:13.971637 (+ 17us) server_negotiation.cc:299] Negotiation successful
0924 04:59:13.971657 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 04:59:13.973557 4879 raft_consensus.cc:1261] T 5fc55278c4f64ae2ae3796e935f383e9 P fbf5946ab1114e85bd81469554f8c893 [term 1 FOLLOWER]: Refusing update from remote peer 075b7df49bf74de8a32a9f3489df31b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:59:13.973573 4746 raft_consensus.cc:1261] T 5fc55278c4f64ae2ae3796e935f383e9 P 70c4cdf67d7b425197eb5609a8ee0f04 [term 1 FOLLOWER]: Refusing update from remote peer 075b7df49bf74de8a32a9f3489df31b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:59:13.973889 4953 consensus_queue.cc:1035] T 5fc55278c4f64ae2ae3796e935f383e9 P 075b7df49bf74de8a32a9f3489df31b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbf5946ab1114e85bd81469554f8c893" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:59:13.974009 4950 consensus_queue.cc:1035] T 5fc55278c4f64ae2ae3796e935f383e9 P 075b7df49bf74de8a32a9f3489df31b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "70c4cdf67d7b425197eb5609a8ee0f04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:59:13.976263 4956 mvcc.cc:205] Tried to move back new op lower bound from 7332765507473047552 to 7332765507300343808. Current Snapshot: MvccSnapshot[applied={T|T < 7332765507473047552}]
I20260924 04:59:13.977270 4957 mvcc.cc:205] Tried to move back new op lower bound from 7332765507473047552 to 7332765507300343808. Current Snapshot: MvccSnapshot[applied={T|T < 7332765507473047552 or (T in {7332765507473047552})}]
I20260924 04:59:13.978533 4468 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54260:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:59:13.978621 4468 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54260:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:59:13.979295 4468 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 924838fa03364764bee727212a018f85: Sending DeleteTablet for 3 replicas of tablet 5fc55278c4f64ae2ae3796e935f383e9
I20260924 04:59:13.979584 4593 tablet_service.cc:1514] Processing DeleteTablet for tablet 5fc55278c4f64ae2ae3796e935f383e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:59:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36034
I20260924 04:59:13.979615 4859 tablet_service.cc:1514] Processing DeleteTablet for tablet 5fc55278c4f64ae2ae3796e935f383e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:59:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33004
I20260924 04:59:13.979928 4726 tablet_service.cc:1514] Processing DeleteTablet for tablet 5fc55278c4f64ae2ae3796e935f383e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:59:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54628
I20260924 04:59:13.980036 4966 tablet_replica.cc:330] T 5fc55278c4f64ae2ae3796e935f383e9 P fbf5946ab1114e85bd81469554f8c893: stopping tablet replica
I20260924 04:59:13.980132 4966 raft_consensus.cc:2229] T 5fc55278c4f64ae2ae3796e935f383e9 P fbf5946ab1114e85bd81469554f8c893 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:59:13.980156 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 4531
I20260924 04:59:13.980240 4966 raft_consensus.cc:2258] T 5fc55278c4f64ae2ae3796e935f383e9 P fbf5946ab1114e85bd81469554f8c893 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:59:13.980737 4968 tablet_replica.cc:330] T 5fc55278c4f64ae2ae3796e935f383e9 P 70c4cdf67d7b425197eb5609a8ee0f04: stopping tablet replica
I20260924 04:59:13.980836 4968 raft_consensus.cc:2229] T 5fc55278c4f64ae2ae3796e935f383e9 P 70c4cdf67d7b425197eb5609a8ee0f04 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:59:13.980927 4968 raft_consensus.cc:2258] T 5fc55278c4f64ae2ae3796e935f383e9 P 70c4cdf67d7b425197eb5609a8ee0f04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:59:13.981613 4968 ts_tablet_manager.cc:1905] T 5fc55278c4f64ae2ae3796e935f383e9 P 70c4cdf67d7b425197eb5609a8ee0f04: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:59:13.982379 4966 ts_tablet_manager.cc:1905] T 5fc55278c4f64ae2ae3796e935f383e9 P fbf5946ab1114e85bd81469554f8c893: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:59:13.983779 4966 ts_tablet_manager.cc:1918] T 5fc55278c4f64ae2ae3796e935f383e9 P fbf5946ab1114e85bd81469554f8c893: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:59:13.983803 4968 ts_tablet_manager.cc:1918] T 5fc55278c4f64ae2ae3796e935f383e9 P 70c4cdf67d7b425197eb5609a8ee0f04: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:59:13.983848 4966 log.cc:1196] T 5fc55278c4f64ae2ae3796e935f383e9 P fbf5946ab1114e85bd81469554f8c893: Deleting WAL directory at /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225929848825-392-0/minicluster-data/ts-2/wal/wals/5fc55278c4f64ae2ae3796e935f383e9
I20260924 04:59:13.983867 4968 log.cc:1196] T 5fc55278c4f64ae2ae3796e935f383e9 P 70c4cdf67d7b425197eb5609a8ee0f04: Deleting WAL directory at /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790225929848825-392-0/minicluster-data/ts-1/wal/wals/5fc55278c4f64ae2ae3796e935f383e9
I20260924 04:59:13.984088 4966 ts_tablet_manager.cc:1939] T 5fc55278c4f64ae2ae3796e935f383e9 P fbf5946ab1114e85bd81469554f8c893: Deleting consensus metadata
I20260924 04:59:13.984171 4968 ts_tablet_manager.cc:1939] T 5fc55278c4f64ae2ae3796e935f383e9 P 70c4cdf67d7b425197eb5609a8ee0f04: Deleting consensus metadata
I20260924 04:59:13.984589 4453 catalog_manager.cc:4841] TS fbf5946ab1114e85bd81469554f8c893 (127.0.98.3:40661): tablet 5fc55278c4f64ae2ae3796e935f383e9 (table test-table [id=7231ee52c2a74f20896da646a350a071]) successfully deleted
I20260924 04:59:13.984671 4454 catalog_manager.cc:4841] TS 70c4cdf67d7b425197eb5609a8ee0f04 (127.0.98.2:44441): tablet 5fc55278c4f64ae2ae3796e935f383e9 (table test-table [id=7231ee52c2a74f20896da646a350a071]) successfully deleted
W20260924 04:59:13.985659 4455 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45753 (error 108)
I20260924 04:59:13.985791 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 4664
I20260924 04:59:13.987112 4937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:13.986983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45753 (local address 127.0.0.1:36056)
0924 04:59:13.987044 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:13.987082 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260924 04:59:13.987236 4455 catalog_manager.cc:4584] TS 075b7df49bf74de8a32a9f3489df31b9 (127.0.98.1:45753): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5fc55278c4f64ae2ae3796e935f383e9: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45753: connect: Connection refused (error 111)
I20260924 04:59:13.990078 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 4797
I20260924 04:59:13.994315 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 4438
2026-09-24T04:59:13Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (1991 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-24T04:59:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:59:14Z Disabled control of system clock
I20260924 04:59:14.020694 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225929848825-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225929848825-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225929848825-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38927
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:39141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38927
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 04:59:14.100256 4973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:59:14.100579 4973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:59:14.100662 4973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:59:14.102262 4973 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:59:14.102330 4973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:59:14.102344 4973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:59:14.102355 4973 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:59:14.102366 4973 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:59:14.103910 4973 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225929848825-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225929848825-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38927
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38927
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4973
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225929848825-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 04:59:14.104223 4973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:59:14.104558 4973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:59:14.109675 4979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:59:14.109751 4973 server_base.cc:1034] running on GCE node
W20260924 04:59:14.109675 4978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:59:14.109697 4981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:59:14.110150 4973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:59:14.110456 4973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:59:14.111624 4973 hybrid_clock.cc:648] HybridClock initialized: now 1790225954111586 us; error 54 us; skew 500 ppm
I20260924 04:59:14.112923 4973 webserver.cc:460] Webserver started at http://127.0.98.62:39019/ using document root <none> and password file <none>
I20260924 04:59:14.113160 4973 fs_manager.cc:359] Metadata directory not provided
I20260924 04:59:14.113209 4973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:59:14.113334 4973 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:59:14.114449 4973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225929848825-392-0/minicluster-data/master-0/data/instance:
uuid: "3f97e9015a95441fb99bf461dd360f60"
format_stamp: "Formatted at 2026-09-24 04:59:14 on dist-test-slave-70st"
server_key: "f4503705dc3758c8f3d382c9ace46393"
server_key_iv: "70f47211a70590b14efdb6a333a35480"
server_key_version: "encryptionkey@0"
I20260924 04:59:14.114796 4973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225929848825-392-0/minicluster-data/master-0/wal/instance:
uuid: "3f97e9015a95441fb99bf461dd360f60"
format_stamp: "Formatted at 2026-09-24 04:59:14 on dist-test-slave-70st"
server_key: "f4503705dc3758c8f3d382c9ace46393"
server_key_iv: "70f47211a70590b14efdb6a333a35480"
server_key_version: "encryptionkey@0"
I20260924 04:59:14.116156 4973 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 04:59:14.116976 4987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:14.117141 4973 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 04:59:14.117214 4973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225929848825-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225929848825-392-0/minicluster-data/master-0/wal
uuid: "3f97e9015a95441fb99bf461dd360f60"
format_stamp: "Formatted at 2026-09-24 04:59:14 on dist-test-slave-70st"
server_key: "f4503705dc3758c8f3d382c9ace46393"
server_key_iv: "70f47211a70590b14efdb6a333a35480"
server_key_version: "encryptionkey@0"
I20260924 04:59:14.117285 4973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225929848825-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225929848825-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225929848825-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:59:14.120642 4973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:59:14.120926 4973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:59:14.121049 4973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:59:14.124732 4973 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38927
I20260924 04:59:14.124773 5039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38927 every 8 connection(s)
I20260924 04:59:14.125093 4973 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:59:14.125799 5040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:59:14.128552 5040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f97e9015a95441fb99bf461dd360f60: Bootstrap starting.
I20260924 04:59:14.129133 5040 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f97e9015a95441fb99bf461dd360f60: Neither blocks nor log segments found. Creating new log.
I20260924 04:59:14.129401 5040 log.cc:824] T 00000000000000000000000000000000 P 3f97e9015a95441fb99bf461dd360f60: Log is configured to *not* fsync() on all Append() calls
I20260924 04:59:14.130103 5040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f97e9015a95441fb99bf461dd360f60: No bootstrap required, opened a new log
I20260924 04:59:14.131775 5040 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f97e9015a95441fb99bf461dd360f60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f97e9015a95441fb99bf461dd360f60" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38927 } }
I20260924 04:59:14.131924 5040 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3f97e9015a95441fb99bf461dd360f60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:59:14.131951 5040 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f97e9015a95441fb99bf461dd360f60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f97e9015a95441fb99bf461dd360f60, State: Initialized, Role: FOLLOWER
I20260924 04:59:14.132102 5040 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f97e9015a95441fb99bf461dd360f60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f97e9015a95441fb99bf461dd360f60" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38927 } }
I20260924 04:59:14.132165 5040 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3f97e9015a95441fb99bf461dd360f60 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:59:14.132185 5040 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f97e9015a95441fb99bf461dd360f60 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:59:14.132212 5040 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f97e9015a95441fb99bf461dd360f60 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:59:14.132742 5040 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f97e9015a95441fb99bf461dd360f60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f97e9015a95441fb99bf461dd360f60" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38927 } }
I20260924 04:59:14.132828 5040 leader_election.cc:302] T 00000000000000000000000000000000 P 3f97e9015a95441fb99bf461dd360f60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f97e9015a95441fb99bf461dd360f60; no voters:
I20260924 04:59:14.132994 5040 leader_election.cc:288] T 00000000000000000000000000000000 P 3f97e9015a95441fb99bf461dd360f60 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:59:14.133049 5043 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f97e9015a95441fb99bf461dd360f60 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:59:14.133229 5043 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f97e9015a95441fb99bf461dd360f60 [term 1 LEADER]: Becoming Leader. State: Replica: 3f97e9015a95441fb99bf461dd360f60, State: Running, Role: LEADER
I20260924 04:59:14.133246 5040 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f97e9015a95441fb99bf461dd360f60 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:59:14.133402 5043 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f97e9015a95441fb99bf461dd360f60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f97e9015a95441fb99bf461dd360f60" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38927 } }
I20260924 04:59:14.133801 5044 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f97e9015a95441fb99bf461dd360f60 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3f97e9015a95441fb99bf461dd360f60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f97e9015a95441fb99bf461dd360f60" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38927 } } }
I20260924 04:59:14.133960 5044 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f97e9015a95441fb99bf461dd360f60 [sys.catalog]: This master's current role is: LEADER
I20260924 04:59:14.134025 5045 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f97e9015a95441fb99bf461dd360f60 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f97e9015a95441fb99bf461dd360f60. Latest consensus state: current_term: 1 leader_uuid: "3f97e9015a95441fb99bf461dd360f60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f97e9015a95441fb99bf461dd360f60" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38927 } } }
I20260924 04:59:14.134083 5045 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f97e9015a95441fb99bf461dd360f60 [sys.catalog]: This master's current role is: LEADER
I20260924 04:59:14.134223 5052 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:59:14.134717 5052 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:59:14.134918 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 4973
I20260924 04:59:14.135021 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225929848825-392-0/minicluster-data/master-0/wal/instance
I20260924 04:59:14.135202 392 external_mini_cluster.cc:1428] Setting key de7a1d2ff61d72e2d9f9a8e386ce49b9
I20260924 04:59:14.136521 5052 catalog_manager.cc:1347] Generated new cluster ID: c3754cb18d1b4902af990ff59752b24b
I20260924 04:59:14.136579 5052 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:59:14.137387 5062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:14.135638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49298 (local address 127.0.98.62:38927)
0924 04:59:14.135806 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:14.135815 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:14.136056 (+ 241us) server_negotiation.cc:407] Connection header received
0924 04:59:14.136196 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:14.136202 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:14.136237 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:14.136377 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:14.136983 (+ 606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:14.136986 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:14.136987 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:59:14.136991 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:14.137032 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:59:14.137043 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:14.137093 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:14.137159 (+ 66us) server_negotiation.cc:299] Negotiation successful
0924 04:59:14.137216 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":67,"threads_started":1}
I20260924 04:59:14.152483 5052 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:59:14.152695 5052 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:59:14.166170 5052 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3f97e9015a95441fb99bf461dd360f60: Generated new TSK 0
I20260924 04:59:14.166381 5052 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:59:14.171205 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225929848825-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225929848825-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225929848825-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225929848825-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38927
--builtin_ntp_servers=127.0.98.20:39141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 04:59:14.256402 5064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:59:14.256676 5064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:59:14.256701 5064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:59:14.257972 5064 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:59:14.258013 5064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:59:14.258038 5064 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:59:14.259407 5064 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225929848825-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225929848825-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225929848825-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38927
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5064
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225929848825-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 04:59:14.259636 5064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:59:14.259944 5064 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:59:14.264860 5070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:59:14.265060 5064 server_base.cc:1034] running on GCE node
W20260924 04:59:14.264858 5069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:59:14.264891 5072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:59:14.265399 5064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:59:14.265679 5064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:59:14.266825 5064 hybrid_clock.cc:648] HybridClock initialized: now 1790225954266808 us; error 35 us; skew 500 ppm
I20260924 04:59:14.268091 5064 webserver.cc:460] Webserver started at http://127.0.98.1:34315/ using document root <none> and password file <none>
I20260924 04:59:14.268312 5064 fs_manager.cc:359] Metadata directory not provided
I20260924 04:59:14.268361 5064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:59:14.268489 5064 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:59:14.269609 5064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225929848825-392-0/minicluster-data/ts-0/data/instance:
uuid: "26d33def70a94fd7bb4dd6466341a15d"
format_stamp: "Formatted at 2026-09-24 04:59:14 on dist-test-slave-70st"
server_key: "e9ea14351dcec595f840a6c863b66424"
server_key_iv: "987ee380e81c21a787c3ab624d6f263a"
server_key_version: "encryptionkey@0"
I20260924 04:59:14.269979 5064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225929848825-392-0/minicluster-data/ts-0/wal/instance:
uuid: "26d33def70a94fd7bb4dd6466341a15d"
format_stamp: "Formatted at 2026-09-24 04:59:14 on dist-test-slave-70st"
server_key: "e9ea14351dcec595f840a6c863b66424"
server_key_iv: "987ee380e81c21a787c3ab624d6f263a"
server_key_version: "encryptionkey@0"
I20260924 04:59:14.271224 5064 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 04:59:14.271915 5078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:14.272079 5064 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:14.272158 5064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225929848825-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225929848825-392-0/minicluster-data/ts-0/wal
uuid: "26d33def70a94fd7bb4dd6466341a15d"
format_stamp: "Formatted at 2026-09-24 04:59:14 on dist-test-slave-70st"
server_key: "e9ea14351dcec595f840a6c863b66424"
server_key_iv: "987ee380e81c21a787c3ab624d6f263a"
server_key_version: "encryptionkey@0"
I20260924 04:59:14.272222 5064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225929848825-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225929848825-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225929848825-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:59:14.275521 5064 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:59:14.275787 5064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:59:14.275925 5064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:59:14.276109 5064 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:59:14.276407 5064 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:59:14.276438 5064 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:14.276458 5064 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:59:14.276476 5064 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:14.281916 5064 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39837
I20260924 04:59:14.281953 5191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39837 every 8 connection(s)
I20260924 04:59:14.282274 5064 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225929848825-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:59:14.285192 5062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:14.282977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45943 (local address 127.0.98.62:38927)
0924 04:59:14.283018 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:14.283020 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:14.283277 (+ 257us) server_negotiation.cc:407] Connection header received
0924 04:59:14.283773 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:14.283775 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:14.283784 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:14.283832 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:14.284953 (+ 1121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:14.284955 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:14.284955 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:59:14.284960 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:14.284976 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:59:14.284980 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:14.285030 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:14.285107 (+ 77us) server_negotiation.cc:299] Negotiation successful
0924 04:59:14.285123 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 04:59:14.285380 5194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:14.282983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38927 (local address 127.0.98.1:45943)
0924 04:59:14.283158 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:14.283191 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:14.283672 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:14.283918 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:14.283922 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:14.283932 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:14.284803 (+ 871us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:59:14.284804 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:14.284832 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:59:14.284863 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:59:14.285032 (+ 169us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:59:14.285037 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:59:14.285040 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 04:59:14.285110 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 04:59:14.285173 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":76,"threads_started":1}
I20260924 04:59:14.285859 5192 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38927
I20260924 04:59:14.285955 5192 heartbeater.cc:464] Registering TS with master...
I20260924 04:59:14.286073 5192 heartbeater.cc:511] Master 127.0.98.62:38927 requested a full tablet report, sending...
I20260924 04:59:14.286430 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 5064
I20260924 04:59:14.286526 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225929848825-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:59:14.286535 5004 ts_manager.cc:194] Registered new tserver with Master: 26d33def70a94fd7bb4dd6466341a15d (127.0.98.1:39837)
I20260924 04:59:14.286701 392 external_mini_cluster.cc:1428] Setting key c3c03e1f37e4efbfd26a8ce2499c4e0e
I20260924 04:59:14.287560 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225929848825-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225929848825-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225929848825-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225929848825-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38927
--builtin_ntp_servers=127.0.98.20:39141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 04:59:14.367851 5195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:59:14.368074 5195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:59:14.368096 5195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:59:14.369230 5195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:59:14.369270 5195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:59:14.369304 5195 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:59:14.370728 5195 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225929848825-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225929848825-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225929848825-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38927
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5195
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225929848825-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 04:59:14.370996 5195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:59:14.371281 5195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:59:14.376210 5203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:59:14.376278 5195 server_base.cc:1034] running on GCE node
W20260924 04:59:14.376210 5200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:59:14.376237 5201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:59:14.376657 5195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:59:14.376891 5195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:59:14.378041 5195 hybrid_clock.cc:648] HybridClock initialized: now 1790225954378029 us; error 28 us; skew 500 ppm
I20260924 04:59:14.379279 5195 webserver.cc:460] Webserver started at http://127.0.98.2:36129/ using document root <none> and password file <none>
I20260924 04:59:14.379524 5195 fs_manager.cc:359] Metadata directory not provided
I20260924 04:59:14.379571 5195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:59:14.379700 5195 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:59:14.380789 5195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225929848825-392-0/minicluster-data/ts-1/data/instance:
uuid: "0cb86753d29d4de59596be7ce570627f"
format_stamp: "Formatted at 2026-09-24 04:59:14 on dist-test-slave-70st"
server_key: "9cb6745dd09674a50a2e5741962fc7b1"
server_key_iv: "4b158e8e5e1a9171879b78a918c820c8"
server_key_version: "encryptionkey@0"
I20260924 04:59:14.381119 5195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225929848825-392-0/minicluster-data/ts-1/wal/instance:
uuid: "0cb86753d29d4de59596be7ce570627f"
format_stamp: "Formatted at 2026-09-24 04:59:14 on dist-test-slave-70st"
server_key: "9cb6745dd09674a50a2e5741962fc7b1"
server_key_iv: "4b158e8e5e1a9171879b78a918c820c8"
server_key_version: "encryptionkey@0"
I20260924 04:59:14.382534 5195 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 04:59:14.383314 5209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:14.383496 5195 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 04:59:14.383579 5195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225929848825-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225929848825-392-0/minicluster-data/ts-1/wal
uuid: "0cb86753d29d4de59596be7ce570627f"
format_stamp: "Formatted at 2026-09-24 04:59:14 on dist-test-slave-70st"
server_key: "9cb6745dd09674a50a2e5741962fc7b1"
server_key_iv: "4b158e8e5e1a9171879b78a918c820c8"
server_key_version: "encryptionkey@0"
I20260924 04:59:14.383642 5195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225929848825-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225929848825-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225929848825-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:59:14.387050 5195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:59:14.387310 5195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:59:14.387429 5195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:59:14.387611 5195 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:59:14.387913 5195 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:59:14.387945 5195 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:14.387972 5195 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:59:14.387995 5195 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:14.393316 5195 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33795
I20260924 04:59:14.393369 5322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33795 every 8 connection(s)
I20260924 04:59:14.393710 5195 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225929848825-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:59:14.396797 5062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:14.394421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54527 (local address 127.0.98.62:38927)
0924 04:59:14.394513 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:14.394515 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:14.394744 (+ 229us) server_negotiation.cc:407] Connection header received
0924 04:59:14.395215 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:14.395217 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:14.395222 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:14.395289 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:14.396417 (+ 1128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:14.396419 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:14.396419 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:59:14.396423 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:14.396441 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:59:14.396446 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:14.396506 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:14.396722 (+ 216us) server_negotiation.cc:299] Negotiation successful
0924 04:59:14.396736 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 04:59:14.397004 5325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:14.394407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38927 (local address 127.0.98.2:54527)
0924 04:59:14.394611 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:14.394652 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:14.395129 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:14.395347 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:14.395350 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:14.395359 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:14.396244 (+ 885us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:59:14.396245 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:14.396274 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:59:14.396306 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:59:14.396613 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:59:14.396619 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:59:14.396622 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 04:59:14.396701 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 04:59:14.396757 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":100,"threads_started":1}
I20260924 04:59:14.397476 5323 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38927
I20260924 04:59:14.397564 5323 heartbeater.cc:464] Registering TS with master...
I20260924 04:59:14.397683 5323 heartbeater.cc:511] Master 127.0.98.62:38927 requested a full tablet report, sending...
I20260924 04:59:14.397996 5004 ts_manager.cc:194] Registered new tserver with Master: 0cb86753d29d4de59596be7ce570627f (127.0.98.2:33795)
I20260924 04:59:14.402005 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 5195
I20260924 04:59:14.402096 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225929848825-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:59:14.402233 392 external_mini_cluster.cc:1428] Setting key b69c5e77fabc5e8f20047d6bbc05ed9b
I20260924 04:59:14.403083 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225929848825-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225929848825-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225929848825-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225929848825-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38927
--builtin_ntp_servers=127.0.98.20:39141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 04:59:14.484117 5326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:59:14.484340 5326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:59:14.484362 5326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:59:14.485549 5326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:59:14.485589 5326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:59:14.485611 5326 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:59:14.486970 5326 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225929848825-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225929848825-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225929848825-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38927
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5326
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225929848825-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 04:59:14.487181 5326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:59:14.487531 5326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:59:14.492295 5331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:59:14.492311 5326 server_base.cc:1034] running on GCE node
W20260924 04:59:14.492277 5332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:59:14.492259 5334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:59:14.492708 5326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:59:14.492954 5326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:59:14.494091 5326 hybrid_clock.cc:648] HybridClock initialized: now 1790225954494078 us; error 27 us; skew 500 ppm
I20260924 04:59:14.495293 5326 webserver.cc:460] Webserver started at http://127.0.98.3:34999/ using document root <none> and password file <none>
I20260924 04:59:14.495520 5326 fs_manager.cc:359] Metadata directory not provided
I20260924 04:59:14.495568 5326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:59:14.495692 5326 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:59:14.496711 5326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225929848825-392-0/minicluster-data/ts-2/data/instance:
uuid: "6c6fa4b2f4034bbfb9cb310c02221509"
format_stamp: "Formatted at 2026-09-24 04:59:14 on dist-test-slave-70st"
server_key: "8522d53b2ee935fcd07310d73ba9fd17"
server_key_iv: "381da896a67f54de94f5c83df0b6be2e"
server_key_version: "encryptionkey@0"
I20260924 04:59:14.497041 5326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225929848825-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6c6fa4b2f4034bbfb9cb310c02221509"
format_stamp: "Formatted at 2026-09-24 04:59:14 on dist-test-slave-70st"
server_key: "8522d53b2ee935fcd07310d73ba9fd17"
server_key_iv: "381da896a67f54de94f5c83df0b6be2e"
server_key_version: "encryptionkey@0"
I20260924 04:59:14.498303 5326 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 04:59:14.498963 5340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:14.499104 5326 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 04:59:14.499189 5326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225929848825-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225929848825-392-0/minicluster-data/ts-2/wal
uuid: "6c6fa4b2f4034bbfb9cb310c02221509"
format_stamp: "Formatted at 2026-09-24 04:59:14 on dist-test-slave-70st"
server_key: "8522d53b2ee935fcd07310d73ba9fd17"
server_key_iv: "381da896a67f54de94f5c83df0b6be2e"
server_key_version: "encryptionkey@0"
I20260924 04:59:14.499248 5326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225929848825-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225929848825-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225929848825-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:59:14.502524 5326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:59:14.502756 5326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:59:14.502871 5326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:59:14.503057 5326 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:59:14.503324 5326 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:59:14.503355 5326 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:14.503381 5326 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:59:14.503404 5326 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:14.508769 5326 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33707
I20260924 04:59:14.508808 5453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33707 every 8 connection(s)
I20260924 04:59:14.509109 5326 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225929848825-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:59:14.511811 5062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:14.509814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39253 (local address 127.0.98.62:38927)
0924 04:59:14.509859 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:14.509861 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:14.510116 (+ 255us) server_negotiation.cc:407] Connection header received
0924 04:59:14.510564 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:14.510565 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:14.510571 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:14.510624 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:14.511627 (+ 1003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:14.511629 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:14.511629 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:59:14.511633 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:14.511647 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:59:14.511651 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:14.511699 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:14.511747 (+ 48us) server_negotiation.cc:299] Negotiation successful
0924 04:59:14.511758 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 04:59:14.511981 5456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:14.509819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38927 (local address 127.0.98.3:39253)
0924 04:59:14.510006 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:14.510038 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:14.510472 (+ 434us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:14.510680 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:14.510683 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:14.510691 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:14.511493 (+ 802us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:59:14.511494 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:14.511522 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:59:14.511547 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:59:14.511696 (+ 149us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:59:14.511698 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:59:14.511700 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 04:59:14.511748 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 04:59:14.511793 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":91,"threads_started":1}
I20260924 04:59:14.512395 5454 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38927
I20260924 04:59:14.512478 5454 heartbeater.cc:464] Registering TS with master...
I20260924 04:59:14.512585 5454 heartbeater.cc:511] Master 127.0.98.62:38927 requested a full tablet report, sending...
I20260924 04:59:14.512879 5004 ts_manager.cc:194] Registered new tserver with Master: 6c6fa4b2f4034bbfb9cb310c02221509 (127.0.98.3:33707)
I20260924 04:59:14.517810 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 5326
I20260924 04:59:14.517920 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225929848825-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:59:14.518069 392 external_mini_cluster.cc:1428] Setting key af08ff1104c31fd6fa593afd1183d73d
I20260924 04:59:14.518893 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:59:14.521145 5062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:14.520135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49314 (local address 127.0.98.62:38927)
0924 04:59:14.520171 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:14.520172 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:14.520299 (+ 127us) server_negotiation.cc:407] Connection header received
0924 04:59:14.520339 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:14.520340 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:14.520345 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:14.520388 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:14.520929 (+ 541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:14.520929 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:14.520930 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:59:14.520933 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:14.520963 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:59:14.520967 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:14.521012 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:14.521068 (+ 56us) server_negotiation.cc:299] Negotiation successful
0924 04:59:14.521079 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 04:59:14.522500 5004 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49314:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:59:14.523916 5004 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:59:14.527331 5462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:14.525818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33795 (local address 127.0.0.1:52028)
0924 04:59:14.526151 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:14.526161 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:14.526207 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:14.526464 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:14.526469 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:14.526476 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:14.526927 (+ 451us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:59:14.526927 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:14.526945 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:59:14.526970 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:59:14.527188 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:59:14.527193 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:59:14.527196 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 04:59:14.527250 (+ 54us) client_negotiation.cc:240] Negotiation successful
0924 04:59:14.527263 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"mutex_wait_us":2,"spinlock_wait_cycles":384,"thread_start_us":82,"threads_started":1}
I20260924 04:59:14.527359 5464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:14.525910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52028 (local address 127.0.98.2:33795)
0924 04:59:14.526317 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:14.526322 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:14.526330 (+ 8us) server_negotiation.cc:407] Connection header received
0924 04:59:14.526354 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:14.526356 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:14.526364 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:14.526444 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:14.527035 (+ 591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:14.527044 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:14.527044 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:59:14.527048 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:14.527083 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:59:14.527112 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:14.527166 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:14.527270 (+ 104us) server_negotiation.cc:299] Negotiation successful
0924 04:59:14.527284 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":82,"threads_started":1}
I20260924 04:59:14.527679 5467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:14.526186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33582 (local address 127.0.98.3:33707)
0924 04:59:14.526364 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:14.526367 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:14.526374 (+ 7us) server_negotiation.cc:407] Connection header received
0924 04:59:14.526394 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:14.526395 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:14.526402 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:14.526460 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:14.527162 (+ 702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:14.527170 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:14.527171 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:59:14.527175 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:14.527207 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:59:14.527234 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:14.527292 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:14.527586 (+ 294us) server_negotiation.cc:299] Negotiation successful
0924 04:59:14.527601 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":89,"threads_started":1}
I20260924 04:59:14.527840 5466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:14.526024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33707 (local address 127.0.0.1:33582)
0924 04:59:14.526203 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:14.526213 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:14.526252 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:14.526685 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:14.526686 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:14.526688 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:14.527063 (+ 375us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:59:14.527063 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:14.527070 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:59:14.527081 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:59:14.527317 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:59:14.527318 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:59:14.527319 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 04:59:14.527364 (+ 45us) client_negotiation.cc:240] Negotiation successful
0924 04:59:14.527375 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":56,"threads_started":1}
I20260924 04:59:14.528118 5463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:14.525818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39837 (local address 127.0.0.1:51526)
0924 04:59:14.525997 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:14.526035 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:14.526087 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:14.526605 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:14.526606 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:14.526609 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:14.527482 (+ 873us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:59:14.527482 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:14.527489 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:59:14.527500 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:59:14.527991 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:59:14.527992 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:59:14.527993 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 04:59:14.528052 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 04:59:14.528064 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":69,"threads_started":1}
I20260924 04:59:14.528004 5257 tablet_service.cc:1467] Processing CreateTablet for tablet 1fc459bfb9214cc4a702d7a7b5f32dd9 (DEFAULT_TABLE table=test-table [id=cef4c2020aae4a018b3febb127648b3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:59:14.528348 5257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fc459bfb9214cc4a702d7a7b5f32dd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:59:14.528469 5388 tablet_service.cc:1467] Processing CreateTablet for tablet 1fc459bfb9214cc4a702d7a7b5f32dd9 (DEFAULT_TABLE table=test-table [id=cef4c2020aae4a018b3febb127648b3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:59:14.528795 5388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fc459bfb9214cc4a702d7a7b5f32dd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:59:14.529551 5465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:14.525984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51526 (local address 127.0.98.1:39837)
0924 04:59:14.526127 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:14.526132 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:14.526142 (+ 10us) server_negotiation.cc:407] Connection header received
0924 04:59:14.526168 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:14.526172 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:14.526182 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:14.526252 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:14.527640 (+ 1388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:14.527644 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:14.527644 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:59:14.527648 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:14.527674 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:59:14.527688 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:14.527747 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:14.528074 (+ 327us) server_negotiation.cc:299] Negotiation successful
0924 04:59:14.529463 (+ 1389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":81,"threads_started":1}
I20260924 04:59:14.530205 5126 tablet_service.cc:1467] Processing CreateTablet for tablet 1fc459bfb9214cc4a702d7a7b5f32dd9 (DEFAULT_TABLE table=test-table [id=cef4c2020aae4a018b3febb127648b3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:59:14.530535 5126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fc459bfb9214cc4a702d7a7b5f32dd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:59:14.531730 5469 tablet_bootstrap.cc:492] T 1fc459bfb9214cc4a702d7a7b5f32dd9 P 6c6fa4b2f4034bbfb9cb310c02221509: Bootstrap starting.
I20260924 04:59:14.531808 5468 tablet_bootstrap.cc:492] T 1fc459bfb9214cc4a702d7a7b5f32dd9 P 0cb86753d29d4de59596be7ce570627f: Bootstrap starting.
I20260924 04:59:14.532248 5469 tablet_bootstrap.cc:654] T 1fc459bfb9214cc4a702d7a7b5f32dd9 P 6c6fa4b2f4034bbfb9cb310c02221509: Neither blocks nor log segments found. Creating new log.
I20260924 04:59:14.532420 5468 tablet_bootstrap.cc:654] T 1fc459bfb9214cc4a702d7a7b5f32dd9 P 0cb86753d29d4de59596be7ce570627f: Neither blocks nor log segments found. Creating new log.
I20260924 04:59:14.532469 5469 log.cc:824] T 1fc459bfb9214cc4a702d7a7b5f32dd9 P 6c6fa4b2f4034bbfb9cb310c02221509: Log is configured to *not* fsync() on all Append() calls
I20260924 04:59:14.532801 5468 log.cc:824] T 1fc459bfb9214cc4a702d7a7b5f32dd9 P 0cb86753d29d4de59596be7ce570627f: Log is configured to *not* fsync() on all Append() calls
I20260924 04:59:14.533053 5469 tablet_bootstrap.cc:492] T 1fc459bfb9214cc4a702d7a7b5f32dd9 P 6c6fa4b2f4034bbfb9cb310c02221509: No bootstrap required, opened a new log
I20260924 04:59:14.533107 5469 ts_tablet_manager.cc:1397] T 1fc459bfb9214cc4a702d7a7b5f32dd9 P 6c6fa4b2f4034bbfb9cb310c02221509: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:59:14.533598 5468 tablet_bootstrap.cc:492] T 1fc459bfb9214cc4a702d7a7b5f32dd9 P 0cb86753d29d4de59596be7ce570627f: No bootstrap required, opened a new log
I20260924 04:59:14.533659 5468 ts_tablet_manager.cc:1397] T 1fc459bfb9214cc4a702d7a7b5f32dd9 P 0cb86753d29d4de59596be7ce570627f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 04:59:14.533962 5472 tablet_bootstrap.cc:492] T 1fc459bfb9214cc4a702d7a7b5f32dd9 P 26d33def70a94fd7bb4dd6466341a15d: Bootstrap starting.
I20260924 04:59:14.534687 5472 tablet_bootstrap.cc:654] T 1fc459bfb9214cc4a702d7a7b5f32dd9 P 26d33def70a94fd7bb4dd6466341a15d: Neither blocks nor log segments found. Creating new log.
I20260924 04:59:14.535028 5472 log.cc:824] T 1fc459bfb9214cc4a702d7a7b5f32dd9 P 26d33def70a94fd7bb4dd6466341a15d: Log is configured to *not* fsync() on all Append() calls
I20260924 04:59:14.535236 5469 raft_consensus.cc:348] T 1fc459bfb9214cc4a702d7a7b5f32dd9 P 6c6fa4b2f4034bbfb9cb310c02221509 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb86753d29d4de59596be7ce570627f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33795 } } peers { permanent_uuid: "26d33def70a94fd7bb4dd6466341a15d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39837 } } peers { permanent_uuid: "6c6fa4b2f4034bbfb9cb310c02221509" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33707 } }
I20260924 04:59:14.535408 5469 raft_consensus.cc:374] T 1fc459bfb9214cc4a702d7a7b5f32dd9 P 6c6fa4b2f4034bbfb9cb310c02221509 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:59:14.535461 5469 raft_consensus.cc:729] T 1fc459bfb9214cc4a702d7a7b5f32dd9 P 6c6fa4b2f4034bbfb9cb310c02221509 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c6fa4b2f4034bbfb9cb310c02221509, State: Initialized, Role: FOLLOWER
I20260924 04:59:14.535578 5469 consensus_queue.cc:260] T 1fc459bfb9214cc4a702d7a7b5f32dd9 P 6c6fa4b2f4034bbfb9cb310c02221509 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb86753d29d4de59596be7ce570627f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33795 } } peers { permanent_uuid: "26d33def70a94fd7bb4dd6466341a15d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39837 } } peers { permanent_uuid: "6c6fa4b2f4034bbfb9cb310c02221509" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33707 } }
I20260924 04:59:14.535809 5469 ts_tablet_manager.cc:1428] T 1fc459bfb9214cc4a702d7a7b5f32dd9 P 6c6fa4b2f4034bbfb9cb310c02221509: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260924 04:59:14.535830 5454 heartbeater.cc:503] Master 127.0.98.62:38927 was elected leader, sending a full tablet report...
I20260924 04:59:14.536012 5472 tablet_bootstrap.cc:492] T 1fc459bfb9214cc4a702d7a7b5f32dd9 P 26d33def70a94fd7bb4dd6466341a15d: No bootstrap required, opened a new log
I20260924 04:59:14.536111 5472 ts_tablet_manager.cc:1397] T 1fc459bfb9214cc4a702d7a7b5f32dd9 P 26d33def70a94fd7bb4dd6466341a15d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 04:59:14.536175 5468 raft_consensus.cc:348] T 1fc459bfb9214cc4a702d7a7b5f32dd9 P 0cb86753d29d4de59596be7ce570627f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb86753d29d4de59596be7ce570627f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33795 } } peers { permanent_uuid: "26d33def70a94fd7bb4dd6466341a15d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39837 } } peers { permanent_uuid: "6c6fa4b2f4034bbfb9cb310c02221509" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33707 } }
I20260924 04:59:14.536329 5468 raft_consensus.cc:374] T 1fc459bfb9214cc4a702d7a7b5f32dd9 P 0cb86753d29d4de59596be7ce570627f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:59:14.536365 5468 raft_consensus.cc:729] T 1fc459bfb9214cc4a702d7a7b5f32dd9 P 0cb86753d29d4de59596be7ce570627f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cb86753d29d4de59596be7ce570627f, State: Initialized, Role: FOLLOWER
I20260924 04:59:14.536504 5468 consensus_queue.cc:260] T 1fc459bfb9214cc4a702d7a7b5f32dd9 P 0cb86753d29d4de59596be7ce570627f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb86753d29d4de59596be7ce570627f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33795 } } peers { permanent_uuid: "26d33def70a94fd7bb4dd6466341a15d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39837 } } peers { permanent_uuid: "6c6fa4b2f4034bbfb9cb310c02221509" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33707 } }
I20260924 04:59:14.536748 5468 ts_tablet_manager.cc:1428] T 1fc459bfb9214cc4a702d7a7b5f32dd9 P 0cb86753d29d4de59596be7ce570627f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 04:59:14.536773 5323 heartbeater.cc:503] Master 127.0.98.62:38927 was elected leader, sending a full tablet report...
I20260924 04:59:14.538523 5472 raft_consensus.cc:348] T 1fc459bfb9214cc4a702d7a7b5f32dd9 P 26d33def70a94fd7bb4dd6466341a15d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb86753d29d4de59596be7ce570627f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33795 } } peers { permanent_uuid: "26d33def70a94fd7bb4dd6466341a15d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39837 } } peers { permanent_uuid: "6c6fa4b2f4034bbfb9cb310c02221509" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33707 } }
I20260924 04:59:14.538647 5472 raft_consensus.cc:374] T 1fc459bfb9214cc4a702d7a7b5f32dd9 P 26d33def70a94fd7bb4dd6466341a15d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:59:14.538667 5472 raft_consensus.cc:729] T 1fc459bfb9214cc4a702d7a7b5f32dd9 P 26d33def70a94fd7bb4dd6466341a15d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26d33def70a94fd7bb4dd6466341a15d, State: Initialized, Role: FOLLOWER
I20260924 04:59:14.538762 5472 consensus_queue.cc:260] T 1fc459bfb9214cc4a702d7a7b5f32dd9 P 26d33def70a94fd7bb4dd6466341a15d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb86753d29d4de59596be7ce570627f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33795 } } peers { permanent_uuid: "26d33def70a94fd7bb4dd6466341a15d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39837 } } peers { permanent_uuid: "6c6fa4b2f4034bbfb9cb310c02221509" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33707 } }
I20260924 04:59:14.538985 5472 ts_tablet_manager.cc:1428] T 1fc459bfb9214cc4a702d7a7b5f32dd9 P 26d33def70a94fd7bb4dd6466341a15d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260924 04:59:14.539031 5192 heartbeater.cc:503] Master 127.0.98.62:38927 was elected leader, sending a full tablet report...
W20260924 04:59:14.644258 5324 tablet.cc:2369] T 1fc459bfb9214cc4a702d7a7b5f32dd9 P 0cb86753d29d4de59596be7ce570627f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 04:59:14.759732 5455 tablet.cc:2369] T 1fc459bfb9214cc4a702d7a7b5f32dd9 P 6c6fa4b2f4034bbfb9cb310c02221509: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 04:59:14.782925 5193 tablet.cc:2369] T 1fc459bfb9214cc4a702d7a7b5f32dd9 P 26d33def70a94fd7bb4dd6466341a15d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:59:14.892602 5474 raft_consensus.cc:482] T 1fc459bfb9214cc4a702d7a7b5f32dd9 P 6c6fa4b2f4034bbfb9cb310c02221509 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:59:14.892724 5474 raft_consensus.cc:504] T 1fc459bfb9214cc4a702d7a7b5f32dd9 P 6c6fa4b2f4034bbfb9cb310c02221509 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb86753d29d4de59596be7ce570627f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33795 } } peers { permanent_uuid: "26d33def70a94fd7bb4dd6466341a15d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39837 } } peers { permanent_uuid: "6c6fa4b2f4034bbfb9cb310c02221509" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33707 } }
I20260924 04:59:14.893042 5474 leader_election.cc:288] T 1fc459bfb9214cc4a702d7a7b5f32dd9 P 6c6fa4b2f4034bbfb9cb310c02221509 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cb86753d29d4de59596be7ce570627f (127.0.98.2:33795), 26d33def70a94fd7bb4dd6466341a15d (127.0.98.1:39837)
I20260924 04:59:14.899577 5475 raft_consensus.cc:482] T 1fc459bfb9214cc4a702d7a7b5f32dd9 P 0cb86753d29d4de59596be7ce570627f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:59:14.899691 5475 raft_consensus.cc:504] T 1fc459bfb9214cc4a702d7a7b5f32dd9 P 0cb86753d29d4de59596be7ce570627f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb86753d29d4de59596be7ce570627f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33795 } } peers { permanent_uuid: "26d33def70a94fd7bb4dd6466341a15d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39837 } } peers { permanent_uuid: "6c6fa4b2f4034bbfb9cb310c02221509" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33707 } }
I20260924 04:59:14.900008 5475 leader_election.cc:288] T 1fc459bfb9214cc4a702d7a7b5f32dd9 P 0cb86753d29d4de59596be7ce570627f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26d33def70a94fd7bb4dd6466341a15d (127.0.98.1:39837), 6c6fa4b2f4034bbfb9cb310c02221509 (127.0.98.3:33707)
I20260924 04:59:14.900792 5456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:14.893240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39837 (local address 127.0.98.3:32995)
0924 04:59:14.893282 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:14.893292 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:14.893330 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:14.893507 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:14.893509 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:14.893512 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:14.900459 (+ 6947us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:59:14.900460 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:14.900469 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:59:14.900487 (+ 18us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:59:14.900661 (+ 174us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:59:14.900661 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:59:14.900662 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 04:59:14.900704 (+ 42us) client_negotiation.cc:240] Negotiation successful
0924 04:59:14.900719 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260924 04:59:14.901049 5465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:14.893304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:32995 (local address 127.0.98.1:39837)
0924 04:59:14.893345 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:14.893347 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:14.893356 (+ 9us) server_negotiation.cc:407] Connection header received
0924 04:59:14.893398 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:14.893399 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:14.893405 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:14.893489 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:14.900555 (+ 7066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:14.900557 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:14.900557 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:59:14.900561 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:14.900578 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:59:14.900583 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:14.900636 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:14.900967 (+ 331us) server_negotiation.cc:299] Negotiation successful
0924 04:59:14.900980 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 04:59:14.901234 5464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:14.893303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58787 (local address 127.0.98.2:33795)
0924 04:59:14.893345 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:14.893347 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:14.893523 (+ 176us) server_negotiation.cc:407] Connection header received
0924 04:59:14.893556 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:14.893557 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:14.893563 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:14.893612 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:14.900595 (+ 6983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:14.900597 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:14.900597 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:59:14.900601 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:14.900616 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:59:14.900620 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:14.900650 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:14.900745 (+ 95us) server_negotiation.cc:299] Negotiation successful
0924 04:59:14.900758 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 04:59:14.901327 5146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fc459bfb9214cc4a702d7a7b5f32dd9" candidate_uuid: "6c6fa4b2f4034bbfb9cb310c02221509" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26d33def70a94fd7bb4dd6466341a15d" is_pre_election: true
I20260924 04:59:14.901502 5467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:14.900289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42179 (local address 127.0.98.3:33707)
0924 04:59:14.900341 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:14.900343 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:14.900440 (+ 97us) server_negotiation.cc:407] Connection header received
0924 04:59:14.900529 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:14.900530 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:14.900535 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:14.900563 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:14.901226 (+ 663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:14.901227 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:14.901227 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:59:14.901232 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:14.901246 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:59:14.901250 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:14.901280 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:14.901393 (+ 113us) server_negotiation.cc:299] Negotiation successful
0924 04:59:14.901406 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 04:59:14.901440 5477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:14.893240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33795 (local address 127.0.98.3:58787)
0924 04:59:14.893457 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:14.893465 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:14.893507 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:14.893632 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:14.893633 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:14.893634 (+ 1us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:14.900467 (+ 6833us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:59:14.900468 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:14.900476 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:59:14.900494 (+ 18us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:59:14.900688 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:59:14.900688 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:59:14.900689 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 04:59:14.901339 (+ 650us) client_negotiation.cc:240] Negotiation successful
0924 04:59:14.901353 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"mutex_wait_us":56,"thread_start_us":93,"threads_started":1}
I20260924 04:59:14.901547 5478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:14.900193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33707 (local address 127.0.98.2:42179)
0924 04:59:14.900387 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:14.900395 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:14.900462 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:14.900628 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:14.900629 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:14.900632 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:14.901092 (+ 460us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:59:14.901093 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:14.901101 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:59:14.901114 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:59:14.901318 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:59:14.901319 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:59:14.901320 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 04:59:14.901462 (+ 142us) client_negotiation.cc:240] Negotiation successful
0924 04:59:14.901475 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":2,"thread_start_us":70,"threads_started":1}
I20260924 04:59:14.901590 5146 raft_consensus.cc:2454] T 1fc459bfb9214cc4a702d7a7b5f32dd9 P 26d33def70a94fd7bb4dd6466341a15d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c6fa4b2f4034bbfb9cb310c02221509 in term 0.
I20260924 04:59:14.901841 5343 leader_election.cc:302] T 1fc459bfb9214cc4a702d7a7b5f32dd9 P 6c6fa4b2f4034bbfb9cb310c02221509 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26d33def70a94fd7bb4dd6466341a15d, 6c6fa4b2f4034bbfb9cb310c02221509; no voters:
I20260924 04:59:14.901903 5277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fc459bfb9214cc4a702d7a7b5f32dd9" candidate_uuid: "6c6fa4b2f4034bbfb9cb310c02221509" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cb86753d29d4de59596be7ce570627f" is_pre_election: true
I20260924 04:59:14.901940 5408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fc459bfb9214cc4a702d7a7b5f32dd9" candidate_uuid: "0cb86753d29d4de59596be7ce570627f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c6fa4b2f4034bbfb9cb310c02221509" is_pre_election: true
I20260924 04:59:14.902000 5474 raft_consensus.cc:2781] T 1fc459bfb9214cc4a702d7a7b5f32dd9 P 6c6fa4b2f4034bbfb9cb310c02221509 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:59:14.902024 5277 raft_consensus.cc:2454] T 1fc459bfb9214cc4a702d7a7b5f32dd9 P 0cb86753d29d4de59596be7ce570627f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c6fa4b2f4034bbfb9cb310c02221509 in term 0.
I20260924 04:59:14.902061 5408 raft_consensus.cc:2454] T 1fc459bfb9214cc4a702d7a7b5f32dd9 P 6c6fa4b2f4034bbfb9cb310c02221509 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cb86753d29d4de59596be7ce570627f in term 0.
I20260924 04:59:14.902117 5474 raft_consensus.cc:482] T 1fc459bfb9214cc4a702d7a7b5f32dd9 P 6c6fa4b2f4034bbfb9cb310c02221509 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:59:14.902153 5474 raft_consensus.cc:3037] T 1fc459bfb9214cc4a702d7a7b5f32dd9 P 6c6fa4b2f4034bbfb9cb310c02221509 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:59:14.902300 5211 leader_election.cc:302] T 1fc459bfb9214cc4a702d7a7b5f32dd9 P 0cb86753d29d4de59596be7ce570627f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cb86753d29d4de59596be7ce570627f, 6c6fa4b2f4034bbfb9cb310c02221509; no voters:
I20260924 04:59:14.902410 5480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:14.900343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43423 (local address 127.0.98.1:39837)
0924 04:59:14.900477 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:14.900479 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:14.900490 (+ 11us) server_negotiation.cc:407] Connection header received
0924 04:59:14.900638 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:14.900640 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:14.900646 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:14.900697 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:14.902085 (+ 1388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:14.902086 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:14.902087 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:59:14.902092 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:14.902138 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:59:14.902144 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:14.902208 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:14.902321 (+ 113us) server_negotiation.cc:299] Negotiation successful
0924 04:59:14.902334 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":53,"threads_started":1}
I20260924 04:59:14.902412 5479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:14.900193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39837 (local address 127.0.98.2:43423)
0924 04:59:14.900313 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:14.900324 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:14.900560 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:14.900746 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:14.900747 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:14.900749 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:14.901818 (+ 1069us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:59:14.901819 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:14.901825 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:59:14.901837 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:59:14.902232 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:59:14.902232 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:59:14.902233 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 04:59:14.902284 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 04:59:14.902296 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":59,"threads_started":1}
I20260924 04:59:14.902424 5475 raft_consensus.cc:2781] T 1fc459bfb9214cc4a702d7a7b5f32dd9 P 0cb86753d29d4de59596be7ce570627f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:59:14.902580 5475 raft_consensus.cc:482] T 1fc459bfb9214cc4a702d7a7b5f32dd9 P 0cb86753d29d4de59596be7ce570627f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:59:14.902601 5475 raft_consensus.cc:3037] T 1fc459bfb9214cc4a702d7a7b5f32dd9 P 0cb86753d29d4de59596be7ce570627f [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:59:14.902683 5146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fc459bfb9214cc4a702d7a7b5f32dd9" candidate_uuid: "0cb86753d29d4de59596be7ce570627f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26d33def70a94fd7bb4dd6466341a15d" is_pre_election: true
I20260924 04:59:14.902760 5146 raft_consensus.cc:2454] T 1fc459bfb9214cc4a702d7a7b5f32dd9 P 26d33def70a94fd7bb4dd6466341a15d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cb86753d29d4de59596be7ce570627f in term 0.
I20260924 04:59:14.903095 5474 raft_consensus.cc:504] T 1fc459bfb9214cc4a702d7a7b5f32dd9 P 6c6fa4b2f4034bbfb9cb310c02221509 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb86753d29d4de59596be7ce570627f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33795 } } peers { permanent_uuid: "26d33def70a94fd7bb4dd6466341a15d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39837 } } peers { permanent_uuid: "6c6fa4b2f4034bbfb9cb310c02221509" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33707 } }
I20260924 04:59:14.903218 5474 leader_election.cc:288] T 1fc459bfb9214cc4a702d7a7b5f32dd9 P 6c6fa4b2f4034bbfb9cb310c02221509 [CANDIDATE]: Term 1 election: Requested vote from peers 0cb86753d29d4de59596be7ce570627f (127.0.98.2:33795), 26d33def70a94fd7bb4dd6466341a15d (127.0.98.1:39837)
I20260924 04:59:14.903348 5277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fc459bfb9214cc4a702d7a7b5f32dd9" candidate_uuid: "6c6fa4b2f4034bbfb9cb310c02221509" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cb86753d29d4de59596be7ce570627f"
I20260924 04:59:14.903409 5146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fc459bfb9214cc4a702d7a7b5f32dd9" candidate_uuid: "6c6fa4b2f4034bbfb9cb310c02221509" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26d33def70a94fd7bb4dd6466341a15d"
I20260924 04:59:14.903492 5146 raft_consensus.cc:3037] T 1fc459bfb9214cc4a702d7a7b5f32dd9 P 26d33def70a94fd7bb4dd6466341a15d [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:59:14.904183 5146 raft_consensus.cc:2454] T 1fc459bfb9214cc4a702d7a7b5f32dd9 P 26d33def70a94fd7bb4dd6466341a15d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c6fa4b2f4034bbfb9cb310c02221509 in term 1.
I20260924 04:59:14.903345 5475 raft_consensus.cc:504] T 1fc459bfb9214cc4a702d7a7b5f32dd9 P 0cb86753d29d4de59596be7ce570627f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb86753d29d4de59596be7ce570627f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33795 } } peers { permanent_uuid: "26d33def70a94fd7bb4dd6466341a15d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39837 } } peers { permanent_uuid: "6c6fa4b2f4034bbfb9cb310c02221509" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33707 } }
I20260924 04:59:14.904330 5343 leader_election.cc:302] T 1fc459bfb9214cc4a702d7a7b5f32dd9 P 6c6fa4b2f4034bbfb9cb310c02221509 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26d33def70a94fd7bb4dd6466341a15d, 6c6fa4b2f4034bbfb9cb310c02221509; no voters:
I20260924 04:59:14.904412 5474 raft_consensus.cc:2781] T 1fc459bfb9214cc4a702d7a7b5f32dd9 P 6c6fa4b2f4034bbfb9cb310c02221509 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:59:14.904403 5475 leader_election.cc:288] T 1fc459bfb9214cc4a702d7a7b5f32dd9 P 0cb86753d29d4de59596be7ce570627f [CANDIDATE]: Term 1 election: Requested vote from peers 26d33def70a94fd7bb4dd6466341a15d (127.0.98.1:39837), 6c6fa4b2f4034bbfb9cb310c02221509 (127.0.98.3:33707)
I20260924 04:59:14.904408 5277 raft_consensus.cc:2379] T 1fc459bfb9214cc4a702d7a7b5f32dd9 P 0cb86753d29d4de59596be7ce570627f [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 6c6fa4b2f4034bbfb9cb310c02221509 in current term 1: Already voted for candidate 0cb86753d29d4de59596be7ce570627f in this term.
I20260924 04:59:14.904567 5474 raft_consensus.cc:686] T 1fc459bfb9214cc4a702d7a7b5f32dd9 P 6c6fa4b2f4034bbfb9cb310c02221509 [term 1 LEADER]: Becoming Leader. State: Replica: 6c6fa4b2f4034bbfb9cb310c02221509, State: Running, Role: LEADER
I20260924 04:59:14.904559 5408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fc459bfb9214cc4a702d7a7b5f32dd9" candidate_uuid: "0cb86753d29d4de59596be7ce570627f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c6fa4b2f4034bbfb9cb310c02221509"
I20260924 04:59:14.904678 5474 consensus_queue.cc:237] T 1fc459bfb9214cc4a702d7a7b5f32dd9 P 6c6fa4b2f4034bbfb9cb310c02221509 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb86753d29d4de59596be7ce570627f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33795 } } peers { permanent_uuid: "26d33def70a94fd7bb4dd6466341a15d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39837 } } peers { permanent_uuid: "6c6fa4b2f4034bbfb9cb310c02221509" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33707 } }
I20260924 04:59:14.905004 5146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fc459bfb9214cc4a702d7a7b5f32dd9" candidate_uuid: "0cb86753d29d4de59596be7ce570627f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26d33def70a94fd7bb4dd6466341a15d"
I20260924 04:59:14.905089 5146 raft_consensus.cc:2379] T 1fc459bfb9214cc4a702d7a7b5f32dd9 P 26d33def70a94fd7bb4dd6466341a15d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 0cb86753d29d4de59596be7ce570627f in current term 1: Already voted for candidate 6c6fa4b2f4034bbfb9cb310c02221509 in this term.
I20260924 04:59:14.905313 5212 leader_election.cc:302] T 1fc459bfb9214cc4a702d7a7b5f32dd9 P 0cb86753d29d4de59596be7ce570627f [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0cb86753d29d4de59596be7ce570627f; no voters: 26d33def70a94fd7bb4dd6466341a15d, 6c6fa4b2f4034bbfb9cb310c02221509
I20260924 04:59:14.905398 5475 raft_consensus.cc:2726] T 1fc459bfb9214cc4a702d7a7b5f32dd9 P 0cb86753d29d4de59596be7ce570627f [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260924 04:59:14.905603 5004 catalog_manager.cc:5495] T 1fc459bfb9214cc4a702d7a7b5f32dd9 P 6c6fa4b2f4034bbfb9cb310c02221509 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6c6fa4b2f4034bbfb9cb310c02221509 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "6c6fa4b2f4034bbfb9cb310c02221509" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb86753d29d4de59596be7ce570627f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26d33def70a94fd7bb4dd6466341a15d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c6fa4b2f4034bbfb9cb310c02221509" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33707 } health_report { overall_health: HEALTHY } } }
I20260924 04:59:14.967450 5467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:14.966471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33596 (local address 127.0.98.3:33707)
0924 04:59:14.966524 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:14.966526 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:14.966545 (+ 19us) server_negotiation.cc:407] Connection header received
0924 04:59:14.966584 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:14.966585 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:14.966599 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:14.966642 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:14.967226 (+ 584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:14.967228 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:14.967228 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:59:14.967232 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:14.967246 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:59:14.967249 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:14.967296 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:14.967345 (+ 49us) server_negotiation.cc:299] Negotiation successful
0924 04:59:14.967359 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 04:59:14.969254 5277 raft_consensus.cc:1261] T 1fc459bfb9214cc4a702d7a7b5f32dd9 P 0cb86753d29d4de59596be7ce570627f [term 1 FOLLOWER]: Refusing update from remote peer 6c6fa4b2f4034bbfb9cb310c02221509: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:59:14.969255 5146 raft_consensus.cc:1261] T 1fc459bfb9214cc4a702d7a7b5f32dd9 P 26d33def70a94fd7bb4dd6466341a15d [term 1 FOLLOWER]: Refusing update from remote peer 6c6fa4b2f4034bbfb9cb310c02221509: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:59:14.969519 5474 consensus_queue.cc:1035] T 1fc459bfb9214cc4a702d7a7b5f32dd9 P 6c6fa4b2f4034bbfb9cb310c02221509 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26d33def70a94fd7bb4dd6466341a15d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:59:14.969671 5481 consensus_queue.cc:1035] T 1fc459bfb9214cc4a702d7a7b5f32dd9 P 6c6fa4b2f4034bbfb9cb310c02221509 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb86753d29d4de59596be7ce570627f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:59:14.972034 5486 mvcc.cc:205] Tried to move back new op lower bound from 7332765511551201280 to 7332765511290134528. Current Snapshot: MvccSnapshot[applied={T|T < 7332765511551201280}]
I20260924 04:59:14.972227 5485 mvcc.cc:205] Tried to move back new op lower bound from 7332765511551201280 to 7332765511290134528. Current Snapshot: MvccSnapshot[applied={T|T < 7332765511551201280}]
I20260924 04:59:14.974622 5003 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49314:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:59:14.974714 5003 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49314:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:59:14.975469 5003 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3f97e9015a95441fb99bf461dd360f60: Sending DeleteTablet for 3 replicas of tablet 1fc459bfb9214cc4a702d7a7b5f32dd9
I20260924 04:59:14.975827 5388 tablet_service.cc:1514] Processing DeleteTablet for tablet 1fc459bfb9214cc4a702d7a7b5f32dd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:59:14 UTC) from {username='slave'} at 127.0.0.1:33582
I20260924 04:59:14.975832 5257 tablet_service.cc:1514] Processing DeleteTablet for tablet 1fc459bfb9214cc4a702d7a7b5f32dd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:59:14 UTC) from {username='slave'} at 127.0.0.1:52028
I20260924 04:59:14.975843 5126 tablet_service.cc:1514] Processing DeleteTablet for tablet 1fc459bfb9214cc4a702d7a7b5f32dd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:59:14 UTC) from {username='slave'} at 127.0.0.1:51526
I20260924 04:59:14.976138 5496 tablet_replica.cc:330] T 1fc459bfb9214cc4a702d7a7b5f32dd9 P 26d33def70a94fd7bb4dd6466341a15d: stopping tablet replica
I20260924 04:59:14.976243 5496 raft_consensus.cc:2229] T 1fc459bfb9214cc4a702d7a7b5f32dd9 P 26d33def70a94fd7bb4dd6466341a15d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:59:14.976343 5496 raft_consensus.cc:2258] T 1fc459bfb9214cc4a702d7a7b5f32dd9 P 26d33def70a94fd7bb4dd6466341a15d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:59:14.976338 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 5064
I20260924 04:59:14.976996 5494 tablet_replica.cc:330] T 1fc459bfb9214cc4a702d7a7b5f32dd9 P 6c6fa4b2f4034bbfb9cb310c02221509: stopping tablet replica
I20260924 04:59:14.977099 5494 raft_consensus.cc:2229] T 1fc459bfb9214cc4a702d7a7b5f32dd9 P 6c6fa4b2f4034bbfb9cb310c02221509 [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:59:14.977186 5494 raft_consensus.cc:2258] T 1fc459bfb9214cc4a702d7a7b5f32dd9 P 6c6fa4b2f4034bbfb9cb310c02221509 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:59:14.977695 5494 ts_tablet_manager.cc:1905] T 1fc459bfb9214cc4a702d7a7b5f32dd9 P 6c6fa4b2f4034bbfb9cb310c02221509: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:59:14.979471 5494 ts_tablet_manager.cc:1918] T 1fc459bfb9214cc4a702d7a7b5f32dd9 P 6c6fa4b2f4034bbfb9cb310c02221509: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:59:14.979583 5494 log.cc:1196] T 1fc459bfb9214cc4a702d7a7b5f32dd9 P 6c6fa4b2f4034bbfb9cb310c02221509: Deleting WAL directory at /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790225929848825-392-0/minicluster-data/ts-2/wal/wals/1fc459bfb9214cc4a702d7a7b5f32dd9
I20260924 04:59:14.979868 5494 ts_tablet_manager.cc:1939] T 1fc459bfb9214cc4a702d7a7b5f32dd9 P 6c6fa4b2f4034bbfb9cb310c02221509: Deleting consensus metadata
I20260924 04:59:14.980314 4989 catalog_manager.cc:4841] TS 6c6fa4b2f4034bbfb9cb310c02221509 (127.0.98.3:33707): tablet 1fc459bfb9214cc4a702d7a7b5f32dd9 (table test-table [id=cef4c2020aae4a018b3febb127648b3b]) successfully deleted
I20260924 04:59:14.981576 5495 tablet_replica.cc:330] T 1fc459bfb9214cc4a702d7a7b5f32dd9 P 0cb86753d29d4de59596be7ce570627f: stopping tablet replica
I20260924 04:59:14.981734 5495 raft_consensus.cc:2229] T 1fc459bfb9214cc4a702d7a7b5f32dd9 P 0cb86753d29d4de59596be7ce570627f [term 1 FOLLOWER]: Raft consensus shutting down.
W20260924 04:59:14.981741 4990 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39837 (error 108)
I20260924 04:59:14.981868 5495 raft_consensus.cc:2258] T 1fc459bfb9214cc4a702d7a7b5f32dd9 P 0cb86753d29d4de59596be7ce570627f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:59:14.981976 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 5195
I20260924 04:59:14.982721 5463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:14.982216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39837 (local address 127.0.0.1:51538)
0924 04:59:14.982630 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:14.982684 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390}
W20260924 04:59:14.982854 4990 catalog_manager.cc:4584] TS 26d33def70a94fd7bb4dd6466341a15d (127.0.98.1:39837): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1fc459bfb9214cc4a702d7a7b5f32dd9: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39837: connect: Connection refused (error 111)
I20260924 04:59:14.986563 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 5326
I20260924 04:59:14.986696 5463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:14.986564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33795 (local address 127.0.0.1:52038)
0924 04:59:14.986624 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:14.986659 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:33795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260924 04:59:14.991339 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 4973
2026-09-24T04:59:14Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (996 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260924 04:59:15.009802 392 mini_oidc.cc:152] Starting JWKS server
I20260924 04:59:15.010816 392 webserver.cc:460] Webserver started at https://127.0.0.1:33797/ using document root <none> and password file <none>
I20260924 04:59:15.010895 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 04:59:15.011053 392 webserver.cc:460] Webserver started at http://127.0.0.1:42121/ using document root <none> and password file <none>
2026-09-24T04:59:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:59:15Z Disabled control of system clock
I20260924 04:59:15.017884 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790225929848825-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790225929848825-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790225929848825-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45037
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:35077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45037
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33797/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790225929848825-392-0/testchainca.cert with env {}
W20260924 04:59:15.100622 5503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:59:15.100823 5503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:59:15.100844 5503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:59:15.102079 5503 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:59:15.102123 5503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:59:15.102134 5503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:59:15.102146 5503 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:59:15.102169 5503 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 04:59:15.102191 5503 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 04:59:15.102205 5503 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33797/jwks.json
I20260924 04:59:15.103732 5503 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790225929848825-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790225929848825-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45037
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45037
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33797/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790225929848825-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5503
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790225929848825-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 04:59:15.103962 5503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:59:15.104161 5503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:59:15.108729 5509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:59:15.108713 5503 server_base.cc:1034] running on GCE node
W20260924 04:59:15.108682 5511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:59:15.108698 5508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:59:15.109161 5503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:59:15.109474 5503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:59:15.110621 5503 hybrid_clock.cc:648] HybridClock initialized: now 1790225955110604 us; error 36 us; skew 500 ppm
I20260924 04:59:15.111821 5503 webserver.cc:460] Webserver started at http://127.0.98.62:34973/ using document root <none> and password file <none>
I20260924 04:59:15.112062 5503 fs_manager.cc:359] Metadata directory not provided
I20260924 04:59:15.112112 5503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:59:15.112234 5503 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:59:15.113262 5503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790225929848825-392-0/minicluster-data/master-0/data/instance:
uuid: "2ffc8cbc551041398272f9f160bc6bec"
format_stamp: "Formatted at 2026-09-24 04:59:15 on dist-test-slave-70st"
server_key: "cba16493f23f3861d867640b700e75fc"
server_key_iv: "107fb1ea4dbedcd2bf652176512d9a3d"
server_key_version: "encryptionkey@0"
I20260924 04:59:15.113619 5503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790225929848825-392-0/minicluster-data/master-0/wal/instance:
uuid: "2ffc8cbc551041398272f9f160bc6bec"
format_stamp: "Formatted at 2026-09-24 04:59:15 on dist-test-slave-70st"
server_key: "cba16493f23f3861d867640b700e75fc"
server_key_iv: "107fb1ea4dbedcd2bf652176512d9a3d"
server_key_version: "encryptionkey@0"
I20260924 04:59:15.114774 5503 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 04:59:15.115437 5517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:15.115581 5503 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 04:59:15.115657 5503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790225929848825-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790225929848825-392-0/minicluster-data/master-0/wal
uuid: "2ffc8cbc551041398272f9f160bc6bec"
format_stamp: "Formatted at 2026-09-24 04:59:15 on dist-test-slave-70st"
server_key: "cba16493f23f3861d867640b700e75fc"
server_key_iv: "107fb1ea4dbedcd2bf652176512d9a3d"
server_key_version: "encryptionkey@0"
I20260924 04:59:15.115723 5503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790225929848825-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790225929848825-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790225929848825-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:59:15.129060 5503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:59:15.129374 5503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:59:15.129534 5503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:59:15.133179 5503 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45037
I20260924 04:59:15.133212 5569 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45037 every 8 connection(s)
I20260924 04:59:15.133545 5503 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:59:15.134469 5570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:59:15.136657 5570 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ffc8cbc551041398272f9f160bc6bec: Bootstrap starting.
I20260924 04:59:15.137181 5570 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2ffc8cbc551041398272f9f160bc6bec: Neither blocks nor log segments found. Creating new log.
I20260924 04:59:15.137404 5570 log.cc:824] T 00000000000000000000000000000000 P 2ffc8cbc551041398272f9f160bc6bec: Log is configured to *not* fsync() on all Append() calls
I20260924 04:59:15.138003 5570 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ffc8cbc551041398272f9f160bc6bec: No bootstrap required, opened a new log
I20260924 04:59:15.139459 5570 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ffc8cbc551041398272f9f160bc6bec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ffc8cbc551041398272f9f160bc6bec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45037 } }
I20260924 04:59:15.139593 5570 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2ffc8cbc551041398272f9f160bc6bec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:59:15.139614 5570 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ffc8cbc551041398272f9f160bc6bec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ffc8cbc551041398272f9f160bc6bec, State: Initialized, Role: FOLLOWER
I20260924 04:59:15.139722 5570 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ffc8cbc551041398272f9f160bc6bec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ffc8cbc551041398272f9f160bc6bec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45037 } }
I20260924 04:59:15.139792 5570 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2ffc8cbc551041398272f9f160bc6bec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:59:15.139819 5570 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ffc8cbc551041398272f9f160bc6bec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:59:15.139850 5570 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ffc8cbc551041398272f9f160bc6bec [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:59:15.140278 5570 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ffc8cbc551041398272f9f160bc6bec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ffc8cbc551041398272f9f160bc6bec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45037 } }
I20260924 04:59:15.140358 5570 leader_election.cc:302] T 00000000000000000000000000000000 P 2ffc8cbc551041398272f9f160bc6bec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ffc8cbc551041398272f9f160bc6bec; no voters:
I20260924 04:59:15.140484 5570 leader_election.cc:288] T 00000000000000000000000000000000 P 2ffc8cbc551041398272f9f160bc6bec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:59:15.140538 5573 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2ffc8cbc551041398272f9f160bc6bec [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:59:15.140699 5573 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2ffc8cbc551041398272f9f160bc6bec [term 1 LEADER]: Becoming Leader. State: Replica: 2ffc8cbc551041398272f9f160bc6bec, State: Running, Role: LEADER
I20260924 04:59:15.140709 5570 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ffc8cbc551041398272f9f160bc6bec [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:59:15.140856 5573 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ffc8cbc551041398272f9f160bc6bec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ffc8cbc551041398272f9f160bc6bec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45037 } }
I20260924 04:59:15.141211 5575 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ffc8cbc551041398272f9f160bc6bec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ffc8cbc551041398272f9f160bc6bec. Latest consensus state: current_term: 1 leader_uuid: "2ffc8cbc551041398272f9f160bc6bec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ffc8cbc551041398272f9f160bc6bec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45037 } } }
I20260924 04:59:15.141294 5574 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ffc8cbc551041398272f9f160bc6bec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2ffc8cbc551041398272f9f160bc6bec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ffc8cbc551041398272f9f160bc6bec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45037 } } }
I20260924 04:59:15.141361 5575 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ffc8cbc551041398272f9f160bc6bec [sys.catalog]: This master's current role is: LEADER
I20260924 04:59:15.141728 5574 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ffc8cbc551041398272f9f160bc6bec [sys.catalog]: This master's current role is: LEADER
I20260924 04:59:15.141808 5582 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:59:15.141827 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 5503
I20260924 04:59:15.141932 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790225929848825-392-0/minicluster-data/master-0/wal/instance
I20260924 04:59:15.142107 392 external_mini_cluster.cc:1428] Setting key e18b4eb9d815124bf24d4e215a245fd6
I20260924 04:59:15.142266 5582 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:59:15.145200 5582 catalog_manager.cc:1347] Generated new cluster ID: 97e705c2a2c74deeb299ba3e64ca777f
I20260924 04:59:15.145254 5582 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:59:15.147413 5592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:15.142519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56734 (local address 127.0.98.62:45037)
0924 04:59:15.142751 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:15.142760 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:15.142793 (+ 33us) server_negotiation.cc:407] Connection header received
0924 04:59:15.142974 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:15.142980 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:15.143045 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:15.143175 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:15.143441 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:15.144767 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:15.145686 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:15.146153 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:15.146267 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:15.146850 (+ 583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:15.146854 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:15.146854 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:59:15.147019 (+ 165us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:15.147051 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:59:15.147061 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:15.147115 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:15.147181 (+ 66us) server_negotiation.cc:299] Negotiation successful
0924 04:59:15.147253 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":72,"threads_started":1}
I20260924 04:59:15.148217 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:59:15.163371 5582 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:59:15.163872 5582 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:59:15.168449 5582 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2ffc8cbc551041398272f9f160bc6bec: Generated new TSK 0
I20260924 04:59:15.168633 5582 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 04:59:15.195796 5600 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260924 04:59:15.195881 5600 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 04:59:15.196177 5592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:15.180677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56746 (local address 127.0.98.62:45037)
0924 04:59:15.180724 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:15.180726 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:15.180804 (+ 78us) server_negotiation.cc:407] Connection header received
0924 04:59:15.180880 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:15.180881 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:15.180893 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:15.180971 (+ 78us) server_negotiation.cc:226] Negotiated authn=JWT
0924 04:59:15.181078 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:15.181888 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:15.182615 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:15.182915 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:15.182974 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:15.183132 (+ 158us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 04:59:15.195951 (+ 12819us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33797/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 04:59:15.196095 (+ 144us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56746: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33797/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":25}
W20260924 04:59:15.196237 5592 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56746: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33797/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260924 04:59:15.196458 5599 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:59:15.180588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45037 (local address 127.0.0.1:56746)
0924 04:59:15.180742 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:15.180752 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:15.180808 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:15.180974 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:15.180974 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:15.180976 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0924 04:59:15.181027 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:15.181029 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:15.181952 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:15.181953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:15.182549 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:15.182552 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:15.182977 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:15.182980 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:15.183024 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:15.183042 (+ 18us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 04:59:15.196063 (+ 13021us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33797/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 04:59:15.196154 (+ 91us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:45037: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33797/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":69,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:45037: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33797/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260924 04:59:15.196947 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 5503
2026-09-24T04:59:15Z chronyd exiting
I20260924 04:59:15.231858 392 test_util.cc:175] -----------------------------------------------
I20260924 04:59:15.231947 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790225929848825-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (223 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260924 04:59:15.232589 392 mini_oidc.cc:152] Starting JWKS server
I20260924 04:59:15.233240 392 webserver.cc:460] Webserver started at https://127.0.0.1:41279/ using document root <none> and password file <none>
I20260924 04:59:15.233278 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 04:59:15.233381 392 webserver.cc:460] Webserver started at http://127.0.0.1:35887/ using document root <none> and password file <none>
2026-09-24T04:59:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:59:15Z Disabled control of system clock
I20260924 04:59:15.239779 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790225929848825-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790225929848825-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790225929848825-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39513
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:35949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39513
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41279/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790225929848825-392-0/testchainca.cert with env {}
W20260924 04:59:15.316079 5606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:59:15.316283 5606 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:59:15.316304 5606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:59:15.317492 5606 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:59:15.317533 5606 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:59:15.317543 5606 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:59:15.317556 5606 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:59:15.317579 5606 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 04:59:15.317602 5606 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 04:59:15.317622 5606 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41279/jwks.json
I20260924 04:59:15.319128 5606 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790225929848825-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790225929848825-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39513
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41279/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790225929848825-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5606
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790225929848825-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 04:59:15.319351 5606 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:59:15.319552 5606 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:59:15.324020 5611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:59:15.323998 5614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:59:15.323971 5612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:59:15.323999 5606 server_base.cc:1034] running on GCE node
I20260924 04:59:15.324435 5606 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:59:15.324677 5606 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:59:15.325811 5606 hybrid_clock.cc:648] HybridClock initialized: now 1790225955325797 us; error 28 us; skew 500 ppm
I20260924 04:59:15.326900 5606 webserver.cc:460] Webserver started at http://127.0.98.62:39955/ using document root <none> and password file <none>
I20260924 04:59:15.327096 5606 fs_manager.cc:359] Metadata directory not provided
I20260924 04:59:15.327138 5606 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:59:15.327255 5606 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:59:15.328238 5606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790225929848825-392-0/minicluster-data/master-0/data/instance:
uuid: "d9d45ddf5f6742ada1ad8d19b2fed346"
format_stamp: "Formatted at 2026-09-24 04:59:15 on dist-test-slave-70st"
server_key: "66b750a0209b0d0123cfdf4eeb2012c0"
server_key_iv: "ff67f7a6213c850a48822e23531f0bd5"
server_key_version: "encryptionkey@0"
I20260924 04:59:15.328563 5606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790225929848825-392-0/minicluster-data/master-0/wal/instance:
uuid: "d9d45ddf5f6742ada1ad8d19b2fed346"
format_stamp: "Formatted at 2026-09-24 04:59:15 on dist-test-slave-70st"
server_key: "66b750a0209b0d0123cfdf4eeb2012c0"
server_key_iv: "ff67f7a6213c850a48822e23531f0bd5"
server_key_version: "encryptionkey@0"
I20260924 04:59:15.329762 5606 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 04:59:15.330415 5620 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:15.330549 5606 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 04:59:15.330617 5606 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790225929848825-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790225929848825-392-0/minicluster-data/master-0/wal
uuid: "d9d45ddf5f6742ada1ad8d19b2fed346"
format_stamp: "Formatted at 2026-09-24 04:59:15 on dist-test-slave-70st"
server_key: "66b750a0209b0d0123cfdf4eeb2012c0"
server_key_iv: "ff67f7a6213c850a48822e23531f0bd5"
server_key_version: "encryptionkey@0"
I20260924 04:59:15.330679 5606 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790225929848825-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790225929848825-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790225929848825-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:59:15.350212 5606 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:59:15.350520 5606 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:59:15.350651 5606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:59:15.354285 5606 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39513
I20260924 04:59:15.354321 5672 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39513 every 8 connection(s)
I20260924 04:59:15.354666 5606 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:59:15.355568 5673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:59:15.357734 5673 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9d45ddf5f6742ada1ad8d19b2fed346: Bootstrap starting.
I20260924 04:59:15.358230 5673 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d9d45ddf5f6742ada1ad8d19b2fed346: Neither blocks nor log segments found. Creating new log.
I20260924 04:59:15.358454 5673 log.cc:824] T 00000000000000000000000000000000 P d9d45ddf5f6742ada1ad8d19b2fed346: Log is configured to *not* fsync() on all Append() calls
I20260924 04:59:15.359021 5673 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9d45ddf5f6742ada1ad8d19b2fed346: No bootstrap required, opened a new log
I20260924 04:59:15.360513 5673 raft_consensus.cc:348] T 00000000000000000000000000000000 P d9d45ddf5f6742ada1ad8d19b2fed346 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9d45ddf5f6742ada1ad8d19b2fed346" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39513 } }
I20260924 04:59:15.360644 5673 raft_consensus.cc:374] T 00000000000000000000000000000000 P d9d45ddf5f6742ada1ad8d19b2fed346 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:59:15.360664 5673 raft_consensus.cc:729] T 00000000000000000000000000000000 P d9d45ddf5f6742ada1ad8d19b2fed346 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9d45ddf5f6742ada1ad8d19b2fed346, State: Initialized, Role: FOLLOWER
I20260924 04:59:15.360766 5673 consensus_queue.cc:260] T 00000000000000000000000000000000 P d9d45ddf5f6742ada1ad8d19b2fed346 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9d45ddf5f6742ada1ad8d19b2fed346" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39513 } }
I20260924 04:59:15.360814 5673 raft_consensus.cc:388] T 00000000000000000000000000000000 P d9d45ddf5f6742ada1ad8d19b2fed346 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:59:15.360832 5673 raft_consensus.cc:482] T 00000000000000000000000000000000 P d9d45ddf5f6742ada1ad8d19b2fed346 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:59:15.360862 5673 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d9d45ddf5f6742ada1ad8d19b2fed346 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:59:15.361292 5673 raft_consensus.cc:504] T 00000000000000000000000000000000 P d9d45ddf5f6742ada1ad8d19b2fed346 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9d45ddf5f6742ada1ad8d19b2fed346" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39513 } }
I20260924 04:59:15.361370 5673 leader_election.cc:302] T 00000000000000000000000000000000 P d9d45ddf5f6742ada1ad8d19b2fed346 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d9d45ddf5f6742ada1ad8d19b2fed346; no voters:
I20260924 04:59:15.361522 5673 leader_election.cc:288] T 00000000000000000000000000000000 P d9d45ddf5f6742ada1ad8d19b2fed346 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:59:15.361562 5676 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d9d45ddf5f6742ada1ad8d19b2fed346 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:59:15.361708 5676 raft_consensus.cc:686] T 00000000000000000000000000000000 P d9d45ddf5f6742ada1ad8d19b2fed346 [term 1 LEADER]: Becoming Leader. State: Replica: d9d45ddf5f6742ada1ad8d19b2fed346, State: Running, Role: LEADER
I20260924 04:59:15.361744 5673 sys_catalog.cc:564] T 00000000000000000000000000000000 P d9d45ddf5f6742ada1ad8d19b2fed346 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:59:15.361804 5676 consensus_queue.cc:237] T 00000000000000000000000000000000 P d9d45ddf5f6742ada1ad8d19b2fed346 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9d45ddf5f6742ada1ad8d19b2fed346" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39513 } }
I20260924 04:59:15.362177 5678 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9d45ddf5f6742ada1ad8d19b2fed346 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d9d45ddf5f6742ada1ad8d19b2fed346. Latest consensus state: current_term: 1 leader_uuid: "d9d45ddf5f6742ada1ad8d19b2fed346" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9d45ddf5f6742ada1ad8d19b2fed346" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39513 } } }
I20260924 04:59:15.362258 5678 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9d45ddf5f6742ada1ad8d19b2fed346 [sys.catalog]: This master's current role is: LEADER
I20260924 04:59:15.362447 5677 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9d45ddf5f6742ada1ad8d19b2fed346 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d9d45ddf5f6742ada1ad8d19b2fed346" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9d45ddf5f6742ada1ad8d19b2fed346" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39513 } } }
I20260924 04:59:15.362471 5685 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:59:15.362706 5677 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9d45ddf5f6742ada1ad8d19b2fed346 [sys.catalog]: This master's current role is: LEADER
I20260924 04:59:15.363130 5685 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:59:15.363492 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 5606
I20260924 04:59:15.363582 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790225929848825-392-0/minicluster-data/master-0/wal/instance
I20260924 04:59:15.363749 392 external_mini_cluster.cc:1428] Setting key 4c9d7a8a0ab1272b09e5f564c10a38ea
I20260924 04:59:15.364748 5685 catalog_manager.cc:1347] Generated new cluster ID: 0cfb249e59574c8c9a6984b75efb3407
I20260924 04:59:15.364799 5685 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:59:15.374886 5696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:15.364342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43644 (local address 127.0.98.62:39513)
0924 04:59:15.369599 (+ 5257us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:15.369608 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:15.369647 (+ 39us) server_negotiation.cc:407] Connection header received
0924 04:59:15.369785 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:15.369794 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:15.369866 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:15.370073 (+ 207us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:15.370286 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:15.371663 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:15.372720 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:15.373385 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:15.373544 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:15.374195 (+ 651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:15.374199 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:15.374200 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:59:15.374410 (+ 210us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:15.374451 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:59:15.374465 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:15.374531 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:15.374585 (+ 54us) server_negotiation.cc:299] Negotiation successful
0924 04:59:15.374660 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5167,"thread_start_us":91,"threads_started":1}
I20260924 04:59:15.375795 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:59:15.380998 5685 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:59:15.381729 5685 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:59:15.396224 5685 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d9d45ddf5f6742ada1ad8d19b2fed346: Generated new TSK 0
I20260924 04:59:15.396441 5685 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 04:59:15.428076 5703 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260924 04:59:15.428177 5703 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 04:59:15.428447 5696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:15.406449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43656 (local address 127.0.98.62:39513)
0924 04:59:15.406496 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:15.406498 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:15.406587 (+ 89us) server_negotiation.cc:407] Connection header received
0924 04:59:15.406665 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:15.406666 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:15.406676 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:15.406733 (+ 57us) server_negotiation.cc:226] Negotiated authn=JWT
0924 04:59:15.406867 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:15.407375 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:15.408493 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:15.408680 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:15.408729 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:15.408899 (+ 170us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 04:59:15.428202 (+ 19303us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41279/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 04:59:15.428367 (+ 165us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43656: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41279/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":26}
W20260924 04:59:15.428510 5696 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43656: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41279/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260924 04:59:15.428556 5702 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:59:15.406348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39513 (local address 127.0.0.1:43656)
0924 04:59:15.406514 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:15.406525 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:15.406594 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:15.406745 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:15.406746 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:15.406748 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0924 04:59:15.406801 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:15.406804 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:15.407459 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:15.407460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:15.408413 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:15.408416 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:15.408741 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:15.408744 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:15.408802 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:15.408808 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 04:59:15.428337 (+ 19529us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41279/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 04:59:15.428460 (+ 123us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:39513: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41279/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":103,"threads_started":1}
I20260924 04:59:15.428841 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 5606
2026-09-24T04:59:15Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (232 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260924 04:59:15.465652 392 mini_oidc.cc:152] Starting JWKS server
I20260924 04:59:15.466310 392 webserver.cc:460] Webserver started at https://127.0.0.1:39505/ using document root <none> and password file <none>
I20260924 04:59:15.466351 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 04:59:15.466470 392 webserver.cc:460] Webserver started at http://127.0.0.1:37171/ using document root <none> and password file <none>
2026-09-24T04:59:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:59:15Z Disabled control of system clock
I20260924 04:59:15.473302 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790225929848825-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790225929848825-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790225929848825-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37375
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:43161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37375
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39505/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 04:59:15.550413 5709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:59:15.550625 5709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:59:15.550653 5709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:59:15.551770 5709 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:59:15.551811 5709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:59:15.551826 5709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:59:15.551841 5709 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:59:15.551856 5709 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 04:59:15.551872 5709 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 04:59:15.551887 5709 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39505/jwks.json
I20260924 04:59:15.553239 5709 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790225929848825-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790225929848825-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37375
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37375
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39505/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5709
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790225929848825-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 04:59:15.553510 5709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:59:15.553782 5709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:59:15.558236 5717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:59:15.558244 5714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:59:15.558312 5715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:59:15.558593 5709 server_base.cc:1034] running on GCE node
I20260924 04:59:15.558816 5709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:59:15.559070 5709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:59:15.560221 5709 hybrid_clock.cc:648] HybridClock initialized: now 1790225955560204 us; error 29 us; skew 500 ppm
I20260924 04:59:15.561399 5709 webserver.cc:460] Webserver started at http://127.0.98.62:43301/ using document root <none> and password file <none>
I20260924 04:59:15.561648 5709 fs_manager.cc:359] Metadata directory not provided
I20260924 04:59:15.561697 5709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:59:15.561821 5709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:59:15.562845 5709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790225929848825-392-0/minicluster-data/master-0/data/instance:
uuid: "00bd1384f6f54526b79d0d1abe428654"
format_stamp: "Formatted at 2026-09-24 04:59:15 on dist-test-slave-70st"
server_key: "3a09cf70d5096e1902192ba4bcc35bfa"
server_key_iv: "47a1d59758ed3999e127141a41147221"
server_key_version: "encryptionkey@0"
I20260924 04:59:15.563171 5709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790225929848825-392-0/minicluster-data/master-0/wal/instance:
uuid: "00bd1384f6f54526b79d0d1abe428654"
format_stamp: "Formatted at 2026-09-24 04:59:15 on dist-test-slave-70st"
server_key: "3a09cf70d5096e1902192ba4bcc35bfa"
server_key_iv: "47a1d59758ed3999e127141a41147221"
server_key_version: "encryptionkey@0"
I20260924 04:59:15.564378 5709 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 04:59:15.565059 5723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:15.565230 5709 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 04:59:15.565310 5709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790225929848825-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790225929848825-392-0/minicluster-data/master-0/wal
uuid: "00bd1384f6f54526b79d0d1abe428654"
format_stamp: "Formatted at 2026-09-24 04:59:15 on dist-test-slave-70st"
server_key: "3a09cf70d5096e1902192ba4bcc35bfa"
server_key_iv: "47a1d59758ed3999e127141a41147221"
server_key_version: "encryptionkey@0"
I20260924 04:59:15.565379 5709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790225929848825-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790225929848825-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790225929848825-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:59:15.581296 5709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:59:15.581646 5709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:59:15.581777 5709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:59:15.585685 5709 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37375
I20260924 04:59:15.585726 5775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37375 every 8 connection(s)
I20260924 04:59:15.586051 5709 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:59:15.587006 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 5709
I20260924 04:59:15.587025 5776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:59:15.587107 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790225929848825-392-0/minicluster-data/master-0/wal/instance
I20260924 04:59:15.587289 392 external_mini_cluster.cc:1428] Setting key 1023e55aff2344332833018e96e971d0
I20260924 04:59:15.589334 5776 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00bd1384f6f54526b79d0d1abe428654: Bootstrap starting.
I20260924 04:59:15.589944 5776 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 00bd1384f6f54526b79d0d1abe428654: Neither blocks nor log segments found. Creating new log.
I20260924 04:59:15.590206 5776 log.cc:824] T 00000000000000000000000000000000 P 00bd1384f6f54526b79d0d1abe428654: Log is configured to *not* fsync() on all Append() calls
I20260924 04:59:15.590955 5776 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00bd1384f6f54526b79d0d1abe428654: No bootstrap required, opened a new log
I20260924 04:59:15.593713 5776 raft_consensus.cc:348] T 00000000000000000000000000000000 P 00bd1384f6f54526b79d0d1abe428654 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00bd1384f6f54526b79d0d1abe428654" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37375 } }
I20260924 04:59:15.593917 5776 raft_consensus.cc:374] T 00000000000000000000000000000000 P 00bd1384f6f54526b79d0d1abe428654 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:59:15.593952 5776 raft_consensus.cc:729] T 00000000000000000000000000000000 P 00bd1384f6f54526b79d0d1abe428654 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00bd1384f6f54526b79d0d1abe428654, State: Initialized, Role: FOLLOWER
I20260924 04:59:15.594076 5776 consensus_queue.cc:260] T 00000000000000000000000000000000 P 00bd1384f6f54526b79d0d1abe428654 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00bd1384f6f54526b79d0d1abe428654" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37375 } }
I20260924 04:59:15.594141 5776 raft_consensus.cc:388] T 00000000000000000000000000000000 P 00bd1384f6f54526b79d0d1abe428654 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:59:15.594170 5776 raft_consensus.cc:482] T 00000000000000000000000000000000 P 00bd1384f6f54526b79d0d1abe428654 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:59:15.594208 5776 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 00bd1384f6f54526b79d0d1abe428654 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:59:15.594931 5776 raft_consensus.cc:504] T 00000000000000000000000000000000 P 00bd1384f6f54526b79d0d1abe428654 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00bd1384f6f54526b79d0d1abe428654" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37375 } }
I20260924 04:59:15.595072 5776 leader_election.cc:302] T 00000000000000000000000000000000 P 00bd1384f6f54526b79d0d1abe428654 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 00bd1384f6f54526b79d0d1abe428654; no voters:
I20260924 04:59:15.595225 5780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:15.590694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46028 (local address 127.0.98.62:37375)
0924 04:59:15.590970 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:15.590977 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:15.591003 (+ 26us) server_negotiation.cc:407] Connection header received
0924 04:59:15.591432 (+ 429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:15.591439 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:15.591496 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:15.591616 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:15.591821 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:15.592719 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:15.593458 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:15.593856 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:15.593973 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:15.594562 (+ 589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:15.594566 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:15.594566 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:59:15.594722 (+ 156us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:15.594753 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:59:15.594763 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:15.594813 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:15.594885 (+ 72us) server_negotiation.cc:299] Negotiation successful
0924 04:59:15.594986 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":63,"threads_started":1}
I20260924 04:59:15.595242 5776 leader_election.cc:288] T 00000000000000000000000000000000 P 00bd1384f6f54526b79d0d1abe428654 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:59:15.595290 5781 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 00bd1384f6f54526b79d0d1abe428654 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:59:15.595391 5781 raft_consensus.cc:686] T 00000000000000000000000000000000 P 00bd1384f6f54526b79d0d1abe428654 [term 1 LEADER]: Becoming Leader. State: Replica: 00bd1384f6f54526b79d0d1abe428654, State: Running, Role: LEADER
I20260924 04:59:15.595477 5776 sys_catalog.cc:564] T 00000000000000000000000000000000 P 00bd1384f6f54526b79d0d1abe428654 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:59:15.595461 5781 consensus_queue.cc:237] T 00000000000000000000000000000000 P 00bd1384f6f54526b79d0d1abe428654 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00bd1384f6f54526b79d0d1abe428654" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37375 } }
I20260924 04:59:15.595779 5781 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00bd1384f6f54526b79d0d1abe428654 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "00bd1384f6f54526b79d0d1abe428654" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00bd1384f6f54526b79d0d1abe428654" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37375 } } }
I20260924 04:59:15.595849 5781 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00bd1384f6f54526b79d0d1abe428654 [sys.catalog]: This master's current role is: LEADER
I20260924 04:59:15.595956 5781 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00bd1384f6f54526b79d0d1abe428654 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 00bd1384f6f54526b79d0d1abe428654. Latest consensus state: current_term: 1 leader_uuid: "00bd1384f6f54526b79d0d1abe428654" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00bd1384f6f54526b79d0d1abe428654" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37375 } } }
I20260924 04:59:15.596007 5781 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00bd1384f6f54526b79d0d1abe428654 [sys.catalog]: This master's current role is: LEADER
I20260924 04:59:15.596076 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260924 04:59:15.598001 5797 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 00bd1384f6f54526b79d0d1abe428654: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:59:15.598047 5797 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:59:15.598088 5785 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:59:15.598243 5785 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:59:15.599687 5785 catalog_manager.cc:1347] Generated new cluster ID: 417fc0f15e8c4219830ea9f6ab4fd194
I20260924 04:59:15.599727 5785 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:59:15.664299 5785 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:59:15.665007 5785 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:59:15.671785 5785 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 00bd1384f6f54526b79d0d1abe428654: Generated new TSK 0
I20260924 04:59:15.671994 5785 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:59:15.687407 5780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:15.683949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46040 (local address 127.0.98.62:37375)
0924 04:59:15.683995 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:15.683997 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:15.684089 (+ 92us) server_negotiation.cc:407] Connection header received
0924 04:59:15.684169 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:15.684169 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:15.684179 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:15.684237 (+ 58us) server_negotiation.cc:226] Negotiated authn=JWT
0924 04:59:15.684391 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:15.684876 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:15.686022 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:15.686220 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:15.686274 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:15.686455 (+ 181us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 04:59:15.687211 (+ 756us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39505/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 04:59:15.687331 (+ 120us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46040: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39505/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":24}
W20260924 04:59:15.687474 5780 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46040: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39505/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260924 04:59:15.687489 5805 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:59:15.683868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37375 (local address 127.0.0.1:46040)
0924 04:59:15.684016 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:15.684028 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:15.684104 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:15.684263 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:15.684264 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:15.684266 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0924 04:59:15.684331 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:15.684333 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:15.684975 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:15.684976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:15.685932 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:15.685935 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:15.686294 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:15.686298 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:15.686359 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:15.686365 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 04:59:15.687300 (+ 935us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39505/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 04:59:15.687387 (+ 87us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:37375: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39505/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":79,"threads_started":1}
I20260924 04:59:15.687834 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 5709
E20260924 04:59:15.698825 5806 webserver.cc:533] Webserver: sslize failed:
E20260924 04:59:15.698904 5806 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-24T04:59:15Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (258 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-24T04:59:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:59:15Z Disabled control of system clock
I20260924 04:59:15.731254 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790225929848825-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790225929848825-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790225929848825-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40807
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:40217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40807
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790225929848825-392-0/testchainca.cert with env {}
W20260924 04:59:15.808868 5810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:59:15.809079 5810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:59:15.809100 5810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:59:15.810253 5810 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:59:15.810292 5810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:59:15.810302 5810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:59:15.810313 5810 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:59:15.810323 5810 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 04:59:15.810334 5810 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 04:59:15.810344 5810 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260924 04:59:15.811800 5810 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790225929848825-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790225929848825-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40807
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40807
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790225929848825-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5810
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790225929848825-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 04:59:15.812031 5810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:59:15.812223 5810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:59:15.816808 5815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:59:15.816767 5816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:59:15.816767 5810 server_base.cc:1034] running on GCE node
W20260924 04:59:15.816879 5818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:59:15.817247 5810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:59:15.817546 5810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:59:15.818684 5810 hybrid_clock.cc:648] HybridClock initialized: now 1790225955818672 us; error 27 us; skew 500 ppm
I20260924 04:59:15.819800 5810 webserver.cc:460] Webserver started at http://127.0.98.62:40003/ using document root <none> and password file <none>
I20260924 04:59:15.820024 5810 fs_manager.cc:359] Metadata directory not provided
I20260924 04:59:15.820070 5810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:59:15.820188 5810 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:59:15.821203 5810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790225929848825-392-0/minicluster-data/master-0/data/instance:
uuid: "f0d095d2e98b43f5abaf3149e55d3d47"
format_stamp: "Formatted at 2026-09-24 04:59:15 on dist-test-slave-70st"
server_key: "13aaccdc3a1339bb6ebd4c4bf3686e7f"
server_key_iv: "6df0f715cc7079fea575c02c32118b05"
server_key_version: "encryptionkey@0"
I20260924 04:59:15.821585 5810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790225929848825-392-0/minicluster-data/master-0/wal/instance:
uuid: "f0d095d2e98b43f5abaf3149e55d3d47"
format_stamp: "Formatted at 2026-09-24 04:59:15 on dist-test-slave-70st"
server_key: "13aaccdc3a1339bb6ebd4c4bf3686e7f"
server_key_iv: "6df0f715cc7079fea575c02c32118b05"
server_key_version: "encryptionkey@0"
I20260924 04:59:15.822749 5810 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 04:59:15.823396 5824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:15.823555 5810 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 04:59:15.823630 5810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790225929848825-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790225929848825-392-0/minicluster-data/master-0/wal
uuid: "f0d095d2e98b43f5abaf3149e55d3d47"
format_stamp: "Formatted at 2026-09-24 04:59:15 on dist-test-slave-70st"
server_key: "13aaccdc3a1339bb6ebd4c4bf3686e7f"
server_key_iv: "6df0f715cc7079fea575c02c32118b05"
server_key_version: "encryptionkey@0"
I20260924 04:59:15.823694 5810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790225929848825-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790225929848825-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790225929848825-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:59:15.833943 5810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:59:15.834210 5810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:59:15.834328 5810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:59:15.837896 5810 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40807
I20260924 04:59:15.837929 5876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40807 every 8 connection(s)
I20260924 04:59:15.838238 5810 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:59:15.839118 5877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:59:15.841289 5877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0d095d2e98b43f5abaf3149e55d3d47: Bootstrap starting.
I20260924 04:59:15.841848 5877 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f0d095d2e98b43f5abaf3149e55d3d47: Neither blocks nor log segments found. Creating new log.
I20260924 04:59:15.842088 5877 log.cc:824] T 00000000000000000000000000000000 P f0d095d2e98b43f5abaf3149e55d3d47: Log is configured to *not* fsync() on all Append() calls
I20260924 04:59:15.842665 5877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0d095d2e98b43f5abaf3149e55d3d47: No bootstrap required, opened a new log
I20260924 04:59:15.844369 5877 raft_consensus.cc:348] T 00000000000000000000000000000000 P f0d095d2e98b43f5abaf3149e55d3d47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0d095d2e98b43f5abaf3149e55d3d47" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40807 } }
I20260924 04:59:15.844538 5877 raft_consensus.cc:374] T 00000000000000000000000000000000 P f0d095d2e98b43f5abaf3149e55d3d47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:59:15.844569 5877 raft_consensus.cc:729] T 00000000000000000000000000000000 P f0d095d2e98b43f5abaf3149e55d3d47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0d095d2e98b43f5abaf3149e55d3d47, State: Initialized, Role: FOLLOWER
I20260924 04:59:15.844705 5877 consensus_queue.cc:260] T 00000000000000000000000000000000 P f0d095d2e98b43f5abaf3149e55d3d47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0d095d2e98b43f5abaf3149e55d3d47" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40807 } }
I20260924 04:59:15.844777 5877 raft_consensus.cc:388] T 00000000000000000000000000000000 P f0d095d2e98b43f5abaf3149e55d3d47 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:59:15.844812 5877 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0d095d2e98b43f5abaf3149e55d3d47 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:59:15.844867 5877 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f0d095d2e98b43f5abaf3149e55d3d47 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:59:15.845198 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 5810
I20260924 04:59:15.845305 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790225929848825-392-0/minicluster-data/master-0/wal/instance
I20260924 04:59:15.845526 392 external_mini_cluster.cc:1428] Setting key 3980e6f61039139144976661d9424455
I20260924 04:59:15.845516 5877 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0d095d2e98b43f5abaf3149e55d3d47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0d095d2e98b43f5abaf3149e55d3d47" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40807 } }
I20260924 04:59:15.845597 5877 leader_election.cc:302] T 00000000000000000000000000000000 P f0d095d2e98b43f5abaf3149e55d3d47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f0d095d2e98b43f5abaf3149e55d3d47; no voters:
I20260924 04:59:15.845724 5877 leader_election.cc:288] T 00000000000000000000000000000000 P f0d095d2e98b43f5abaf3149e55d3d47 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:59:15.845759 5880 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f0d095d2e98b43f5abaf3149e55d3d47 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:59:15.845883 5880 raft_consensus.cc:686] T 00000000000000000000000000000000 P f0d095d2e98b43f5abaf3149e55d3d47 [term 1 LEADER]: Becoming Leader. State: Replica: f0d095d2e98b43f5abaf3149e55d3d47, State: Running, Role: LEADER
I20260924 04:59:15.845966 5877 sys_catalog.cc:564] T 00000000000000000000000000000000 P f0d095d2e98b43f5abaf3149e55d3d47 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:59:15.845966 5880 consensus_queue.cc:237] T 00000000000000000000000000000000 P f0d095d2e98b43f5abaf3149e55d3d47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0d095d2e98b43f5abaf3149e55d3d47" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40807 } }
I20260924 04:59:15.847759 5882 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0d095d2e98b43f5abaf3149e55d3d47 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f0d095d2e98b43f5abaf3149e55d3d47" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0d095d2e98b43f5abaf3149e55d3d47" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40807 } } }
I20260924 04:59:15.847918 5881 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0d095d2e98b43f5abaf3149e55d3d47 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f0d095d2e98b43f5abaf3149e55d3d47. Latest consensus state: current_term: 1 leader_uuid: "f0d095d2e98b43f5abaf3149e55d3d47" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0d095d2e98b43f5abaf3149e55d3d47" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40807 } } }
W20260924 04:59:15.848078 5896 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f0d095d2e98b43f5abaf3149e55d3d47: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260924 04:59:15.848106 5882 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0d095d2e98b43f5abaf3149e55d3d47 [sys.catalog]: This master's current role is: LEADER
I20260924 04:59:15.848116 5881 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0d095d2e98b43f5abaf3149e55d3d47 [sys.catalog]: This master's current role is: LEADER
W20260924 04:59:15.848119 5896 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:59:15.848291 5899 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:59:15.848464 5899 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:59:15.850914 5899 catalog_manager.cc:1347] Generated new cluster ID: bad7b1a1fa0446c2813959376c4f963c
I20260924 04:59:15.850968 5899 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:59:15.853137 5894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:15.846819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49660 (local address 127.0.98.62:40807)
0924 04:59:15.847417 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:15.847426 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:15.847461 (+ 35us) server_negotiation.cc:407] Connection header received
0924 04:59:15.847582 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:15.847589 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:15.847661 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:15.848315 (+ 654us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:15.848447 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:15.849805 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:15.850621 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:15.851231 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:15.852110 (+ 879us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:15.852265 (+ 155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:15.852268 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:15.852269 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:59:15.852472 (+ 203us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:15.852513 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:59:15.852527 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:15.852818 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:15.852850 (+ 32us) server_negotiation.cc:299] Negotiation successful
0924 04:59:15.852928 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":456,"threads_started":1}
I20260924 04:59:15.854070 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 04:59:15.862744 5899 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:59:15.863207 5899 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:59:15.867271 5899 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f0d095d2e98b43f5abaf3149e55d3d47: Generated new TSK 0
I20260924 04:59:15.867439 5899 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:59:16.310814 5894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:15.875379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49662 (local address 127.0.98.62:40807)
0924 04:59:15.875435 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:15.875437 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:15.875493 (+ 56us) server_negotiation.cc:407] Connection header received
0924 04:59:15.875562 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:15.875562 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:15.875571 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:15.875627 (+ 56us) server_negotiation.cc:226] Negotiated authn=JWT
0924 04:59:15.875749 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:15.876241 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:15.877307 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:15.877529 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:15.877575 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:15.877746 (+ 171us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 04:59:16.310590 (+432844us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 04:59:16.310750 (+ 160us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49662: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":30}
W20260924 04:59:16.310878 5894 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49662: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260924 04:59:16.310963 5906 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:59:15.875292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40807 (local address 127.0.0.1:49662)
0924 04:59:15.875430 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:15.875441 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:15.875505 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:15.875636 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:15.875637 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:15.875639 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0924 04:59:15.875696 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:15.875699 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:15.876305 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:15.876306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:15.877238 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:15.877241 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:15.877596 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:15.877599 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:15.877664 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:15.877669 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 04:59:16.310729 (+433060us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 04:59:16.310852 (+ 123us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:40807: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":66,"threads_started":1}
I20260924 04:59:16.311316 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 5810
2026-09-24T04:59:16Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (603 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (86 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (84 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225929848825-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:59:16 dist-test-slave-70st krb5kdc[5918](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:59:16 dist-test-slave-70st krb5kdc[5918](info): set up 2 sockets
Sep 24 04:59:16 dist-test-slave-70st krb5kdc[5918](info): commencing operation
krb5kdc: starting...
W20260924 04:59:17.433534 392 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225929848825-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225929848825-392-0/krb5kdc/test-user.keytab.
Sep 24 04:59:17 dist-test-slave-70st krb5kdc[5918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225957, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:59:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:59:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:59:17.546429 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225929848825-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225929848825-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225929848825-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36479
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36479
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:59:17.625447 5931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:59:17.625670 5931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:59:17.625701 5931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:59:17.626924 5931 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:59:17.626969 5931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:59:17.626988 5931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:59:17.627005 5931 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:59:17.627020 5931 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:59:17.628610 5931 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225929848825-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225929848825-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36479
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36479
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5931
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225929848825-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 04:59:17.628871 5931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:59:17.629087 5931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:59:17.633796 5936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:59:17.633806 5937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:59:17.633903 5939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:59:17.633957 5931 server_base.cc:1034] running on GCE node
I20260924 04:59:17.634214 5931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:59:17.634516 5931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:59:17.635669 5931 hybrid_clock.cc:648] HybridClock initialized: now 1790225957635653 us; error 35 us; skew 500 ppm
Sep 24 04:59:17 dist-test-slave-70st krb5kdc[5918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225957, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:59:17.638154 5931 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:59:17.638628 5931 webserver.cc:460] Webserver started at http://127.0.98.62:39707/ using document root <none> and password file <none>
I20260924 04:59:17.638839 5931 fs_manager.cc:359] Metadata directory not provided
I20260924 04:59:17.638890 5931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:59:17.639021 5931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:59:17.640112 5931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225929848825-392-0/minicluster-data/master-0/data/instance:
uuid: "af6c67716c924fc1983dad17a20e14a4"
format_stamp: "Formatted at 2026-09-24 04:59:17 on dist-test-slave-70st"
server_key: "e37fb1f9110753e13e60ac4268b99727"
server_key_iv: "00a3f67096ab5bd670da852976855afe"
server_key_version: "encryptionkey@0"
I20260924 04:59:17.640467 5931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225929848825-392-0/minicluster-data/master-0/wal/instance:
uuid: "af6c67716c924fc1983dad17a20e14a4"
format_stamp: "Formatted at 2026-09-24 04:59:17 on dist-test-slave-70st"
server_key: "e37fb1f9110753e13e60ac4268b99727"
server_key_iv: "00a3f67096ab5bd670da852976855afe"
server_key_version: "encryptionkey@0"
I20260924 04:59:17.641875 5931 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260924 04:59:17.642663 5946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:17.642835 5931 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 04:59:17.642923 5931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225929848825-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225929848825-392-0/minicluster-data/master-0/wal
uuid: "af6c67716c924fc1983dad17a20e14a4"
format_stamp: "Formatted at 2026-09-24 04:59:17 on dist-test-slave-70st"
server_key: "e37fb1f9110753e13e60ac4268b99727"
server_key_iv: "00a3f67096ab5bd670da852976855afe"
server_key_version: "encryptionkey@0"
I20260924 04:59:17.642992 5931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225929848825-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225929848825-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225929848825-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:59:17.655212 5931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:59:17.657104 5931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:59:17.657253 5931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:59:17.661490 5931 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36479
I20260924 04:59:17.661517 5998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36479 every 8 connection(s)
I20260924 04:59:17.662022 5931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:59:17.662738 5999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:59:17.665028 5999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af6c67716c924fc1983dad17a20e14a4: Bootstrap starting.
I20260924 04:59:17.665611 5999 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P af6c67716c924fc1983dad17a20e14a4: Neither blocks nor log segments found. Creating new log.
I20260924 04:59:17.665863 5999 log.cc:824] T 00000000000000000000000000000000 P af6c67716c924fc1983dad17a20e14a4: Log is configured to *not* fsync() on all Append() calls
I20260924 04:59:17.666481 5999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af6c67716c924fc1983dad17a20e14a4: No bootstrap required, opened a new log
I20260924 04:59:17.668030 5999 raft_consensus.cc:348] T 00000000000000000000000000000000 P af6c67716c924fc1983dad17a20e14a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af6c67716c924fc1983dad17a20e14a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36479 } }
I20260924 04:59:17.668172 5999 raft_consensus.cc:374] T 00000000000000000000000000000000 P af6c67716c924fc1983dad17a20e14a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:59:17.668196 5999 raft_consensus.cc:729] T 00000000000000000000000000000000 P af6c67716c924fc1983dad17a20e14a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af6c67716c924fc1983dad17a20e14a4, State: Initialized, Role: FOLLOWER
I20260924 04:59:17.668308 5999 consensus_queue.cc:260] T 00000000000000000000000000000000 P af6c67716c924fc1983dad17a20e14a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af6c67716c924fc1983dad17a20e14a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36479 } }
I20260924 04:59:17.668375 5999 raft_consensus.cc:388] T 00000000000000000000000000000000 P af6c67716c924fc1983dad17a20e14a4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:59:17.668397 5999 raft_consensus.cc:482] T 00000000000000000000000000000000 P af6c67716c924fc1983dad17a20e14a4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:59:17.668435 5999 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af6c67716c924fc1983dad17a20e14a4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:59:17.669274 5999 raft_consensus.cc:504] T 00000000000000000000000000000000 P af6c67716c924fc1983dad17a20e14a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af6c67716c924fc1983dad17a20e14a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36479 } }
I20260924 04:59:17.669366 5999 leader_election.cc:302] T 00000000000000000000000000000000 P af6c67716c924fc1983dad17a20e14a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: af6c67716c924fc1983dad17a20e14a4; no voters:
I20260924 04:59:17.669551 5999 leader_election.cc:288] T 00000000000000000000000000000000 P af6c67716c924fc1983dad17a20e14a4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:59:17.669603 6002 raft_consensus.cc:2781] T 00000000000000000000000000000000 P af6c67716c924fc1983dad17a20e14a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:59:17.669791 5999 sys_catalog.cc:564] T 00000000000000000000000000000000 P af6c67716c924fc1983dad17a20e14a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:59:17.669804 6002 raft_consensus.cc:686] T 00000000000000000000000000000000 P af6c67716c924fc1983dad17a20e14a4 [term 1 LEADER]: Becoming Leader. State: Replica: af6c67716c924fc1983dad17a20e14a4, State: Running, Role: LEADER
I20260924 04:59:17.669968 6002 consensus_queue.cc:237] T 00000000000000000000000000000000 P af6c67716c924fc1983dad17a20e14a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af6c67716c924fc1983dad17a20e14a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36479 } }
I20260924 04:59:17.670315 6004 sys_catalog.cc:455] T 00000000000000000000000000000000 P af6c67716c924fc1983dad17a20e14a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader af6c67716c924fc1983dad17a20e14a4. Latest consensus state: current_term: 1 leader_uuid: "af6c67716c924fc1983dad17a20e14a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af6c67716c924fc1983dad17a20e14a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36479 } } }
I20260924 04:59:17.670465 6003 sys_catalog.cc:455] T 00000000000000000000000000000000 P af6c67716c924fc1983dad17a20e14a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "af6c67716c924fc1983dad17a20e14a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af6c67716c924fc1983dad17a20e14a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36479 } } }
I20260924 04:59:17.670536 6004 sys_catalog.cc:458] T 00000000000000000000000000000000 P af6c67716c924fc1983dad17a20e14a4 [sys.catalog]: This master's current role is: LEADER
I20260924 04:59:17.670610 6003 sys_catalog.cc:458] T 00000000000000000000000000000000 P af6c67716c924fc1983dad17a20e14a4 [sys.catalog]: This master's current role is: LEADER
I20260924 04:59:17.670918 6011 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:59:17.671029 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 5931
I20260924 04:59:17.671147 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225929848825-392-0/minicluster-data/master-0/wal/instance
I20260924 04:59:17.671341 392 external_mini_cluster.cc:1428] Setting key c9559bd33b2d79cb144a86684293bd0d
I20260924 04:59:17.671378 6011 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:59:17.672940 6011 catalog_manager.cc:1347] Generated new cluster ID: 1d80ab2f91824dd8b789dfd53c4cd77b
I20260924 04:59:17.672994 6011 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 04:59:17 dist-test-slave-70st krb5kdc[5918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225957, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:59:17.680348 6021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:17.671976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48520 (local address 127.0.98.62:36479)
0924 04:59:17.672131 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:17.672140 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:17.672218 (+ 78us) server_negotiation.cc:407] Connection header received
0924 04:59:17.672360 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:17.672367 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:17.672424 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:17.672551 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:17.673199 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:17.674153 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:17.674852 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:17.675308 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:17.675401 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:17.677534 (+ 2133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:17.677537 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:17.677538 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:59:17.677549 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:17.679013 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:17.679323 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:17.679326 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:17.679327 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:17.679421 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:17.679586 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:17.679587 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:17.679587 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:17.679814 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:17.679959 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:17.680032 (+ 73us) server_negotiation.cc:299] Negotiation successful
0924 04:59:17.680160 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":74,"threads_started":1}
I20260924 04:59:17.691490 6011 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:59:17.692028 6011 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:59:17.705323 6011 catalog_manager.cc:5868] T 00000000000000000000000000000000 P af6c67716c924fc1983dad17a20e14a4: Generated new TSK 0
I20260924 04:59:17.705541 6011 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225929848825-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225929848825-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:59:17.734172 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225929848825-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225929848825-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225929848825-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225929848825-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36479
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225929848825-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:59:17.812804 6025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:59:17.813015 6025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:59:17.813038 6025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:59:17.814177 6025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:59:17.814209 6025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:59:17.814234 6025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:59:17.815665 6025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225929848825-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225929848825-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225929848825-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225929848825-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36479
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6025
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225929848825-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 04:59:17.815866 6025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:59:17.816042 6025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:59:17.820811 6033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:59:17.820804 6031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:59:17.821074 6025 server_base.cc:1034] running on GCE node
W20260924 04:59:17.820804 6030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:59:17.821324 6025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:59:17.821583 6025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:59:17.822733 6025 hybrid_clock.cc:648] HybridClock initialized: now 1790225957822718 us; error 30 us; skew 500 ppm
Sep 24 04:59:17 dist-test-slave-70st krb5kdc[5918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225957, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:59:17.825117 6025 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:59:17.825640 6025 webserver.cc:460] Webserver started at http://127.0.98.1:42513/ using document root <none> and password file <none>
I20260924 04:59:17.825846 6025 fs_manager.cc:359] Metadata directory not provided
I20260924 04:59:17.825893 6025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:59:17.826012 6025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:59:17.827006 6025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225929848825-392-0/minicluster-data/ts-0/data/instance:
uuid: "06bc100ba3934d54ae9619bffa3353d5"
format_stamp: "Formatted at 2026-09-24 04:59:17 on dist-test-slave-70st"
server_key: "ca4b405693433736faff60c91ba86cb2"
server_key_iv: "ae516e15fb9ed0d7f944b46fd37107f8"
server_key_version: "encryptionkey@0"
I20260924 04:59:17.827327 6025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225929848825-392-0/minicluster-data/ts-0/wal/instance:
uuid: "06bc100ba3934d54ae9619bffa3353d5"
format_stamp: "Formatted at 2026-09-24 04:59:17 on dist-test-slave-70st"
server_key: "ca4b405693433736faff60c91ba86cb2"
server_key_iv: "ae516e15fb9ed0d7f944b46fd37107f8"
server_key_version: "encryptionkey@0"
I20260924 04:59:17.828516 6025 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 04:59:17.829154 6040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:17.829311 6025 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 04:59:17.829391 6025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225929848825-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225929848825-392-0/minicluster-data/ts-0/wal
uuid: "06bc100ba3934d54ae9619bffa3353d5"
format_stamp: "Formatted at 2026-09-24 04:59:17 on dist-test-slave-70st"
server_key: "ca4b405693433736faff60c91ba86cb2"
server_key_iv: "ae516e15fb9ed0d7f944b46fd37107f8"
server_key_version: "encryptionkey@0"
I20260924 04:59:17.829499 6025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225929848825-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225929848825-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225929848825-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:59:17.841336 6025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:59:17.843505 6025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:59:17.843643 6025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:59:17.843852 6025 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:59:17.844151 6025 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:59:17.844184 6025 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:17.844213 6025 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:59:17.844236 6025 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:17.849687 6025 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46297
I20260924 04:59:17.849742 6153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46297 every 8 connection(s)
I20260924 04:59:17.850035 6025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225929848825-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 04:59:17 dist-test-slave-70st krb5kdc[5918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225957, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:59:17.858732 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 6025
I20260924 04:59:17.858841 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225929848825-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:59:17.859036 392 external_mini_cluster.cc:1428] Setting key e0616a7cb9691d1cd0d54ae331824698
I20260924 04:59:17.859778 6021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:17.850761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59631 (local address 127.0.98.62:36479)
0924 04:59:17.850839 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:17.850841 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:17.850982 (+ 141us) server_negotiation.cc:407] Connection header received
0924 04:59:17.852029 (+ 1047us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:17.852030 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:17.852047 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:17.852137 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:17.852953 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:17.853718 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:17.854789 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:17.855002 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:17.855058 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:17.857112 (+ 2054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:17.857113 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:17.857113 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:59:17.857121 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:17.858326 (+ 1205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:17.858740 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:17.858741 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:17.858741 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:17.858805 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:17.859132 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:17.859133 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:17.859134 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:17.859246 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:17.859329 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:17.859610 (+ 281us) server_negotiation.cc:299] Negotiation successful
0924 04:59:17.859699 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 04:59:17.859997 6156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:17.850713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36479 (local address 127.0.98.1:59631)
0924 04:59:17.850861 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:17.850903 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:17.851920 (+ 1017us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:17.852232 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:17.852236 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:17.852588 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:17.852855 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:17.852861 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:17.853813 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:17.853814 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:17.854714 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:17.854717 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:17.855074 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:17.855075 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:17.855192 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:17.855693 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:17.855702 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:17.857022 (+ 1320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:59:17.858400 (+ 1378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:17.858407 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:17.858410 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:17.858661 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:17.858921 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:17.858922 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:17.858923 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:17.859066 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:17.859325 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:59:17.859332 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:59:17.859483 (+ 151us) client_negotiation.cc:769] Sending connection context
0924 04:59:17.859584 (+ 101us) client_negotiation.cc:240] Negotiation successful
0924 04:59:17.859750 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":78,"threads_started":1}
I20260924 04:59:17.860464 6154 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36479
I20260924 04:59:17.860558 6154 heartbeater.cc:464] Registering TS with master...
I20260924 04:59:17.860740 6154 heartbeater.cc:511] Master 127.0.98.62:36479 requested a full tablet report, sending...
I20260924 04:59:17.861316 5963 ts_manager.cc:194] Registered new tserver with Master: 06bc100ba3934d54ae9619bffa3353d5 (127.0.98.1:46297)
I20260924 04:59:17.862185 5963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:59631
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225929848825-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225929848825-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:59:17.882422 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225929848825-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225929848825-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225929848825-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225929848825-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36479
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225929848825-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:59:17.960642 6159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:59:17.960860 6159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:59:17.960882 6159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:59:17.962044 6159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:59:17.962086 6159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:59:17.962112 6159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:59:17.963569 6159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225929848825-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225929848825-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225929848825-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225929848825-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36479
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6159
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225929848825-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 04:59:17.963802 6159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:59:17.964018 6159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:59:17.968888 6164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:59:17.968911 6167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:59:17.969004 6159 server_base.cc:1034] running on GCE node
W20260924 04:59:17.968912 6165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:59:17.969362 6159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:59:17.969640 6159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:59:17.970775 6159 hybrid_clock.cc:648] HybridClock initialized: now 1790225957970753 us; error 40 us; skew 500 ppm
Sep 24 04:59:17 dist-test-slave-70st krb5kdc[5918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225957, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:59:17.973214 6159 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:59:17.973673 6159 webserver.cc:460] Webserver started at http://127.0.98.2:41709/ using document root <none> and password file <none>
I20260924 04:59:17.973872 6159 fs_manager.cc:359] Metadata directory not provided
I20260924 04:59:17.973919 6159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:59:17.974037 6159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:59:17.975049 6159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225929848825-392-0/minicluster-data/ts-1/data/instance:
uuid: "4d96be73b8b04ab2b3ee30e33fc8b9fe"
format_stamp: "Formatted at 2026-09-24 04:59:17 on dist-test-slave-70st"
server_key: "e998164941a5bb1367919d233da23611"
server_key_iv: "ac0707e521a4060e99681a5df4ff4c6f"
server_key_version: "encryptionkey@0"
I20260924 04:59:17.975379 6159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225929848825-392-0/minicluster-data/ts-1/wal/instance:
uuid: "4d96be73b8b04ab2b3ee30e33fc8b9fe"
format_stamp: "Formatted at 2026-09-24 04:59:17 on dist-test-slave-70st"
server_key: "e998164941a5bb1367919d233da23611"
server_key_iv: "ac0707e521a4060e99681a5df4ff4c6f"
server_key_version: "encryptionkey@0"
I20260924 04:59:17.976555 6159 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 04:59:17.977231 6174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:17.977366 6159 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 04:59:17.977465 6159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225929848825-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225929848825-392-0/minicluster-data/ts-1/wal
uuid: "4d96be73b8b04ab2b3ee30e33fc8b9fe"
format_stamp: "Formatted at 2026-09-24 04:59:17 on dist-test-slave-70st"
server_key: "e998164941a5bb1367919d233da23611"
server_key_iv: "ac0707e521a4060e99681a5df4ff4c6f"
server_key_version: "encryptionkey@0"
I20260924 04:59:17.977530 6159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225929848825-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225929848825-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225929848825-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:59:17.987453 6159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:59:17.989605 6159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:59:17.989740 6159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:59:17.989964 6159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:59:17.990254 6159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:59:17.990301 6159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:17.990330 6159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:59:17.990355 6159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:17.995784 6159 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45351
I20260924 04:59:17.995844 6287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45351 every 8 connection(s)
I20260924 04:59:17.996141 6159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225929848825-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:59:17.996675 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 6159
I20260924 04:59:17.996774 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225929848825-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:59:17.996954 392 external_mini_cluster.cc:1428] Setting key c3b23c636b8f91394dbbb70917881c3b
Sep 24 04:59:17 dist-test-slave-70st krb5kdc[5918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225957, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 04:59:18.005499 6021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:17.996855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59791 (local address 127.0.98.62:36479)
0924 04:59:17.996905 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:17.996907 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:17.997130 (+ 223us) server_negotiation.cc:407] Connection header received
0924 04:59:17.998158 (+ 1028us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:17.998159 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:17.998171 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:17.998227 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:17.998873 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:17.999367 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:18.000327 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:18.000527 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:18.000585 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:18.002879 (+ 2294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:18.002880 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:18.002881 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:59:18.002888 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:18.004171 (+ 1283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:18.004524 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:18.004525 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:18.004525 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:18.004576 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:18.004788 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:18.004789 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:18.004789 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:18.004916 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:18.004983 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:18.005301 (+ 318us) server_negotiation.cc:299] Negotiation successful
0924 04:59:18.005377 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 04:59:18.005654 6290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:17.996851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36479 (local address 127.0.98.2:59791)
0924 04:59:17.996997 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:17.997037 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:17.998056 (+ 1019us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:17.998308 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:17.998312 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:17.998564 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:17.998765 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:17.998771 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:17.999454 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:17.999455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:18.000237 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:18.000239 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:18.000600 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:18.000601 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:18.000715 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:18.001179 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:18.001187 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:18.002789 (+ 1602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:59:18.004239 (+ 1450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:18.004245 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:18.004247 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:18.004459 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:18.004627 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:18.004628 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:18.004628 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:18.004729 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:18.005002 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:59:18.005007 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:59:18.005139 (+ 132us) client_negotiation.cc:769] Sending connection context
0924 04:59:18.005250 (+ 111us) client_negotiation.cc:240] Negotiation successful
0924 04:59:18.005412 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":77,"threads_started":1}
I20260924 04:59:18.006136 6288 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36479
I20260924 04:59:18.006243 6288 heartbeater.cc:464] Registering TS with master...
I20260924 04:59:18.006449 6288 heartbeater.cc:511] Master 127.0.98.62:36479 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260924 04:59:18.006819 5963 ts_manager.cc:194] Registered new tserver with Master: 4d96be73b8b04ab2b3ee30e33fc8b9fe (127.0.98.2:45351)
I20260924 04:59:18.007181 5963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:59791
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225929848825-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225929848825-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:59:18.021117 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225929848825-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225929848825-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225929848825-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225929848825-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36479
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225929848825-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:59:18.102234 6293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:59:18.102439 6293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:59:18.102463 6293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:59:18.103641 6293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:59:18.103721 6293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:59:18.103755 6293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:59:18.105271 6293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225929848825-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225929848825-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225929848825-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225929848825-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36479
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6293
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225929848825-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 04:59:18.105509 6293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:59:18.105722 6293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:59:18.110515 6293 server_base.cc:1034] running on GCE node
W20260924 04:59:18.110496 6301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:59:18.110492 6298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:59:18.110492 6299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:59:18.110905 6293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:59:18.111133 6293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:59:18.112264 6293 hybrid_clock.cc:648] HybridClock initialized: now 1790225958112254 us; error 29 us; skew 500 ppm
Sep 24 04:59:18 dist-test-slave-70st krb5kdc[5918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225958, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:59:18.114662 6293 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:59:18.115088 6293 webserver.cc:460] Webserver started at http://127.0.98.3:46183/ using document root <none> and password file <none>
I20260924 04:59:18.115281 6293 fs_manager.cc:359] Metadata directory not provided
I20260924 04:59:18.115329 6293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:59:18.115432 6293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:59:18.116439 6293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225929848825-392-0/minicluster-data/ts-2/data/instance:
uuid: "c310f9b0e1f142e6954400ceb9a22412"
format_stamp: "Formatted at 2026-09-24 04:59:18 on dist-test-slave-70st"
server_key: "365b74257972595d6637343e4479f257"
server_key_iv: "9d16c3a6136a27aa021c0d2bf69ba4dc"
server_key_version: "encryptionkey@0"
I20260924 04:59:18.116750 6293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225929848825-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c310f9b0e1f142e6954400ceb9a22412"
format_stamp: "Formatted at 2026-09-24 04:59:18 on dist-test-slave-70st"
server_key: "365b74257972595d6637343e4479f257"
server_key_iv: "9d16c3a6136a27aa021c0d2bf69ba4dc"
server_key_version: "encryptionkey@0"
I20260924 04:59:18.118001 6293 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 04:59:18.118604 6308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:18.118753 6293 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 04:59:18.118830 6293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225929848825-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225929848825-392-0/minicluster-data/ts-2/wal
uuid: "c310f9b0e1f142e6954400ceb9a22412"
format_stamp: "Formatted at 2026-09-24 04:59:18 on dist-test-slave-70st"
server_key: "365b74257972595d6637343e4479f257"
server_key_iv: "9d16c3a6136a27aa021c0d2bf69ba4dc"
server_key_version: "encryptionkey@0"
I20260924 04:59:18.118887 6293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225929848825-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225929848825-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225929848825-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:59:18.125015 6293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:59:18.126991 6293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:59:18.127105 6293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:59:18.127275 6293 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:59:18.127517 6293 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:59:18.127545 6293 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:18.127564 6293 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:59:18.127575 6293 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:18.133002 6293 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33647
I20260924 04:59:18.133060 6421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33647 every 8 connection(s)
I20260924 04:59:18.133334 6293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225929848825-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:59:18.135747 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 6293
I20260924 04:59:18.135847 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225929848825-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:59:18.136030 392 external_mini_cluster.cc:1428] Setting key 1c715e0f535873774c1d1e146e53d87d
Sep 24 04:59:18 dist-test-slave-70st krb5kdc[5918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225958, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:59:18.142987 6021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:18.134080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44387 (local address 127.0.98.62:36479)
0924 04:59:18.134154 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:18.134156 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:18.134394 (+ 238us) server_negotiation.cc:407] Connection header received
0924 04:59:18.135337 (+ 943us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:18.135339 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:18.135349 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:18.135412 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:18.136382 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:18.137011 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:18.138079 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:18.138286 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:18.138346 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:18.140398 (+ 2052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:18.140400 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:18.140400 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:59:18.140408 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:18.141651 (+ 1243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:18.142027 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:18.142028 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:18.142029 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:18.142097 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:18.142324 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:18.142325 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:18.142325 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:18.142449 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:18.142511 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:18.142808 (+ 297us) server_negotiation.cc:299] Negotiation successful
0924 04:59:18.142891 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 04:59:18.143146 6424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:18.134041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36479 (local address 127.0.98.3:44387)
0924 04:59:18.134205 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:18.134281 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:18.135208 (+ 927us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:18.135497 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:18.135501 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:18.135865 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:18.136146 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:18.136153 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:18.137093 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:18.137094 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:18.137991 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:18.137994 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:18.138368 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:18.138369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:18.138496 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:18.138940 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:18.138948 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:18.140309 (+ 1361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:59:18.141720 (+ 1411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:18.141727 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:18.141729 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:18.141949 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:18.142152 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:18.142153 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:18.142153 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:18.142265 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:18.142506 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:59:18.142512 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:59:18.142671 (+ 159us) client_negotiation.cc:769] Sending connection context
0924 04:59:18.142777 (+ 106us) client_negotiation.cc:240] Negotiation successful
0924 04:59:18.142933 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":84,"threads_started":1}
I20260924 04:59:18.143584 6422 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36479
I20260924 04:59:18.143679 6422 heartbeater.cc:464] Registering TS with master...
I20260924 04:59:18.143831 6422 heartbeater.cc:511] Master 127.0.98.62:36479 requested a full tablet report, sending...
I20260924 04:59:18.144182 5963 ts_manager.cc:194] Registered new tserver with Master: c310f9b0e1f142e6954400ceb9a22412 (127.0.98.3:33647)
I20260924 04:59:18.144531 5963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:44387
I20260924 04:59:18.147720 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225929848825-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225929848825-392-0/krb5kdc/test-admin.keytab.
Sep 24 04:59:18 dist-test-slave-70st krb5kdc[5918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:59:18.162784 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260924 04:59:18.164296 6429 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:59:18.163628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43702 (local address 127.0.98.1:46297)
0924 04:59:18.163773 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:18.163777 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:18.163787 (+ 10us) server_negotiation.cc:407] Connection header received
0924 04:59:18.163812 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:18.163814 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:18.163827 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:18.163900 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:18.164215 (+ 315us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43702: BlockingRecv error: recv got EOF from 127.0.0.1:43702 (error 108)
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":71,"threads_started":1}
W20260924 04:59:18.164434 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225929848825-392-0/krb5kdc/krb5cc' of type FILE
Sep 24 04:59:18 dist-test-slave-70st krb5kdc[5918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:59:18.165443 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 04:59:18 dist-test-slave-70st krb5kdc[5918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:59:18.166247 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 04:59:18 dist-test-slave-70st krb5kdc[5918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 04:59:18.174126 6429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:18.166773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43714 (local address 127.0.98.1:46297)
0924 04:59:18.166820 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:18.166821 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:18.166857 (+ 36us) server_negotiation.cc:407] Connection header received
0924 04:59:18.166925 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:18.166925 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:18.166936 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:18.166973 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:18.167299 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:18.168042 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:18.168910 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:18.169303 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:18.169402 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:18.171590 (+ 2188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:18.171599 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:18.171599 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:59:18.171606 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:18.173102 (+ 1496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:18.173393 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:18.173395 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:18.173395 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:18.173500 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:18.173649 (+ 149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:18.173650 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:18.173650 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:18.173775 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:18.173873 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:18.173956 (+ 83us) server_negotiation.cc:299] Negotiation successful
0924 04:59:18.174021 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
W20260924 04:59:18.175781 6429 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:59:18.175324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43724 (local address 127.0.98.1:46297)
0924 04:59:18.175361 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:18.175363 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:18.175446 (+ 83us) server_negotiation.cc:407] Connection header received
0924 04:59:18.175483 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:18.175484 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:18.175494 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:18.175535 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:18.175744 (+ 209us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43724: BlockingRecv error: recv got EOF from 127.0.0.1:43724 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260924 04:59:18.175967 392 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225929848825-392-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225929848825-392-0/krb5kdc/krb5cc)
Sep 24 04:59:18 dist-test-slave-70st krb5kdc[5918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:59:18.176908 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 04:59:18 dist-test-slave-70st krb5kdc[5918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:59:18.177696 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 04:59:18 dist-test-slave-70st krb5kdc[5918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 04:59:18.184724 6429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:18.178198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43732 (local address 127.0.98.1:46297)
0924 04:59:18.178246 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:18.178247 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:18.178272 (+ 25us) server_negotiation.cc:407] Connection header received
0924 04:59:18.178321 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:18.178322 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:18.178328 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:18.178368 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:18.178659 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:18.179143 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:18.179811 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:18.180059 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:18.180106 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:18.182308 (+ 2202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:18.182309 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:18.182310 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:59:18.182316 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:18.183555 (+ 1239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:18.183956 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:18.183957 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:18.183957 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:18.184020 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:18.184234 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:18.184234 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:18.184235 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:18.184349 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:18.184443 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:18.184548 (+ 105us) server_negotiation.cc:299] Negotiation successful
0924 04:59:18.184626 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
W20260924 04:59:18.186167 6429 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:59:18.185685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43738 (local address 127.0.98.1:46297)
0924 04:59:18.185725 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:18.185727 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:18.185771 (+ 44us) server_negotiation.cc:407] Connection header received
0924 04:59:18.185826 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:18.185827 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:18.185835 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:18.185877 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:18.186121 (+ 244us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43738: BlockingRecv error: recv got EOF from 127.0.0.1:43738 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260924 04:59:18.186218 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790225929848825-392-0/krb5kdc/krb5cc' of type FILE
Sep 24 04:59:18 dist-test-slave-70st krb5kdc[5918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:59:18.187134 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 04:59:18 dist-test-slave-70st krb5kdc[5918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:59:18.187888 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 04:59:18 dist-test-slave-70st krb5kdc[5918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 04:59:18.194069 6429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:18.188356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43754 (local address 127.0.98.1:46297)
0924 04:59:18.188402 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:18.188404 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:18.188413 (+ 9us) server_negotiation.cc:407] Connection header received
0924 04:59:18.188476 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:18.188476 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:18.188484 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:18.188526 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:18.188842 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:18.189303 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:18.189998 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:18.190186 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:18.190239 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:18.192067 (+ 1828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:18.192069 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:18.192069 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:59:18.192076 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:18.193102 (+ 1026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:18.193381 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:18.193382 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:18.193382 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:18.193478 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:18.193638 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:18.193638 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:18.193639 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:18.193729 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:18.193818 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:18.193904 (+ 86us) server_negotiation.cc:299] Negotiation successful
0924 04:59:18.193974 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 04:59:18.194625 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 6025
I20260924 04:59:18.199218 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 6159
I20260924 04:59:18.202781 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 6293
I20260924 04:59:18.206424 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 5931
2026-09-24T04:59:18Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1726 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:59:18 dist-test-slave-70st krb5kdc[6442](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:59:18 dist-test-slave-70st krb5kdc[6442](info): set up 2 sockets
Sep 24 04:59:18 dist-test-slave-70st krb5kdc[6442](info): commencing operation
krb5kdc: starting...
W20260924 04:59:19.161639 392 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/krb5kdc/test-user.keytab.
Sep 24 04:59:19 dist-test-slave-70st krb5kdc[6442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225959, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:59:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:59:19Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260924 04:59:19.275801 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43351
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43351
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:59:19.353248 6455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:59:19.353492 6455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:59:19.353523 6455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:59:19.354645 6455 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:59:19.354686 6455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:59:19.354703 6455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:59:19.354719 6455 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:59:19.354734 6455 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:59:19.356163 6455 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43351
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43351
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6455
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 04:59:19.356392 6455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:59:19.356588 6455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:59:19.361089 6460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:59:19.361094 6463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:59:19.361094 6461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:59:19.361111 6455 server_base.cc:1034] running on GCE node
I20260924 04:59:19.361573 6455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:59:19.361871 6455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:59:19.363059 6455 hybrid_clock.cc:648] HybridClock initialized: now 1790225959363042 us; error 36 us; skew 500 ppm
Sep 24 04:59:19 dist-test-slave-70st krb5kdc[6442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225959, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:59:19.365664 6455 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260924 04:59:19.366096 6455 webserver.cc:460] Webserver started at http://127.0.98.62:45305/ using document root <none> and password file <none>
I20260924 04:59:19.366292 6455 fs_manager.cc:359] Metadata directory not provided
I20260924 04:59:19.366346 6455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:59:19.366468 6455 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:59:19.367549 6455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/master-0/data/instance:
uuid: "221d34b534b046078e9003aff8289b32"
format_stamp: "Formatted at 2026-09-24 04:59:19 on dist-test-slave-70st"
server_key: "41d8cdd1096251d04b29d466c6f6a397"
server_key_iv: "a85621b4ca05cfa79ee5ee2aece24361"
server_key_version: "encryptionkey@0"
I20260924 04:59:19.367935 6455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/master-0/wal/instance:
uuid: "221d34b534b046078e9003aff8289b32"
format_stamp: "Formatted at 2026-09-24 04:59:19 on dist-test-slave-70st"
server_key: "41d8cdd1096251d04b29d466c6f6a397"
server_key_iv: "a85621b4ca05cfa79ee5ee2aece24361"
server_key_version: "encryptionkey@0"
I20260924 04:59:19.369184 6455 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 04:59:19.369997 6470 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:19.370198 6455 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 04:59:19.370280 6455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/master-0/wal
uuid: "221d34b534b046078e9003aff8289b32"
format_stamp: "Formatted at 2026-09-24 04:59:19 on dist-test-slave-70st"
server_key: "41d8cdd1096251d04b29d466c6f6a397"
server_key_iv: "a85621b4ca05cfa79ee5ee2aece24361"
server_key_version: "encryptionkey@0"
I20260924 04:59:19.370342 6455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:59:19.390358 6455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:59:19.392170 6455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:59:19.392300 6455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:59:19.396044 6455 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43351
I20260924 04:59:19.396078 6522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43351 every 8 connection(s)
I20260924 04:59:19.396402 6455 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:59:19.397328 6523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:59:19.399544 6523 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 221d34b534b046078e9003aff8289b32: Bootstrap starting.
I20260924 04:59:19.400089 6523 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 221d34b534b046078e9003aff8289b32: Neither blocks nor log segments found. Creating new log.
I20260924 04:59:19.400265 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 6455
I20260924 04:59:19.400372 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/master-0/wal/instance
I20260924 04:59:19.400404 6523 log.cc:824] T 00000000000000000000000000000000 P 221d34b534b046078e9003aff8289b32: Log is configured to *not* fsync() on all Append() calls
I20260924 04:59:19.400558 392 external_mini_cluster.cc:1428] Setting key 6bf2e7fb23487bfa6103fe4cecdc89bd
I20260924 04:59:19.401132 6523 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 221d34b534b046078e9003aff8289b32: No bootstrap required, opened a new log
I20260924 04:59:19.403513 6523 raft_consensus.cc:348] T 00000000000000000000000000000000 P 221d34b534b046078e9003aff8289b32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "221d34b534b046078e9003aff8289b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43351 } }
I20260924 04:59:19.403685 6523 raft_consensus.cc:374] T 00000000000000000000000000000000 P 221d34b534b046078e9003aff8289b32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:59:19.403721 6523 raft_consensus.cc:729] T 00000000000000000000000000000000 P 221d34b534b046078e9003aff8289b32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 221d34b534b046078e9003aff8289b32, State: Initialized, Role: FOLLOWER
I20260924 04:59:19.403841 6523 consensus_queue.cc:260] T 00000000000000000000000000000000 P 221d34b534b046078e9003aff8289b32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "221d34b534b046078e9003aff8289b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43351 } }
I20260924 04:59:19.403916 6523 raft_consensus.cc:388] T 00000000000000000000000000000000 P 221d34b534b046078e9003aff8289b32 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:59:19.403949 6523 raft_consensus.cc:482] T 00000000000000000000000000000000 P 221d34b534b046078e9003aff8289b32 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:59:19.403992 6523 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 221d34b534b046078e9003aff8289b32 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:59:19.404595 6523 raft_consensus.cc:504] T 00000000000000000000000000000000 P 221d34b534b046078e9003aff8289b32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "221d34b534b046078e9003aff8289b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43351 } }
I20260924 04:59:19.404680 6523 leader_election.cc:302] T 00000000000000000000000000000000 P 221d34b534b046078e9003aff8289b32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 221d34b534b046078e9003aff8289b32; no voters:
I20260924 04:59:19.404819 6523 leader_election.cc:288] T 00000000000000000000000000000000 P 221d34b534b046078e9003aff8289b32 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:59:19.404856 6528 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 221d34b534b046078e9003aff8289b32 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:59:19.404992 6528 raft_consensus.cc:686] T 00000000000000000000000000000000 P 221d34b534b046078e9003aff8289b32 [term 1 LEADER]: Becoming Leader. State: Replica: 221d34b534b046078e9003aff8289b32, State: Running, Role: LEADER
I20260924 04:59:19.405061 6523 sys_catalog.cc:564] T 00000000000000000000000000000000 P 221d34b534b046078e9003aff8289b32 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:59:19.405130 6528 consensus_queue.cc:237] T 00000000000000000000000000000000 P 221d34b534b046078e9003aff8289b32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "221d34b534b046078e9003aff8289b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43351 } }
I20260924 04:59:19.405481 6528 sys_catalog.cc:455] T 00000000000000000000000000000000 P 221d34b534b046078e9003aff8289b32 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 221d34b534b046078e9003aff8289b32. Latest consensus state: current_term: 1 leader_uuid: "221d34b534b046078e9003aff8289b32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "221d34b534b046078e9003aff8289b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43351 } } }
I20260924 04:59:19.405558 6528 sys_catalog.cc:458] T 00000000000000000000000000000000 P 221d34b534b046078e9003aff8289b32 [sys.catalog]: This master's current role is: LEADER
I20260924 04:59:19.405638 6530 sys_catalog.cc:455] T 00000000000000000000000000000000 P 221d34b534b046078e9003aff8289b32 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "221d34b534b046078e9003aff8289b32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "221d34b534b046078e9003aff8289b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43351 } } }
I20260924 04:59:19.405787 6530 sys_catalog.cc:458] T 00000000000000000000000000000000 P 221d34b534b046078e9003aff8289b32 [sys.catalog]: This master's current role is: LEADER
I20260924 04:59:19.406113 6537 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:59:19.406567 6537 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 24 04:59:19 dist-test-slave-70st krb5kdc[6442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225959, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 04:59:19.408478 6537 catalog_manager.cc:1347] Generated new cluster ID: c56b85dc80bd450c8c3816861b8b9cb0
I20260924 04:59:19.408524 6537 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:59:19.412508 6527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:19.401049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34276 (local address 127.0.98.62:43351)
0924 04:59:19.401205 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:19.401215 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:19.401250 (+ 35us) server_negotiation.cc:407] Connection header received
0924 04:59:19.401711 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:19.401719 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:19.401768 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:19.401892 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:19.402608 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:19.403490 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:19.404477 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:19.404967 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:19.405376 (+ 409us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:19.408622 (+ 3246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:19.408626 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:19.408626 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:59:19.408638 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:19.410513 (+ 1875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:19.410913 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:19.410916 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:19.410916 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:19.411030 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:19.411226 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:19.411226 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:19.411227 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:19.411522 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:19.411662 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:19.412166 (+ 504us) server_negotiation.cc:299] Negotiation successful
0924 04:59:19.412311 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":70,"threads_started":1}
I20260924 04:59:19.419049 6537 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:59:19.419571 6537 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:59:19.421780 6537 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 221d34b534b046078e9003aff8289b32: Generated new TSK 0
I20260924 04:59:19.421932 6537 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260924 04:59:19.448745 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43351
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:59:19.526459 6549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:59:19.526656 6549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:59:19.526678 6549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:59:19.527732 6549 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:59:19.527766 6549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:59:19.527799 6549 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:59:19.529210 6549 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43351
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6549
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 04:59:19.529397 6549 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:59:19.529601 6549 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:59:19.534117 6557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:59:19.534194 6554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:59:19.534299 6549 server_base.cc:1034] running on GCE node
W20260924 04:59:19.534349 6555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:59:19.534538 6549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:59:19.534770 6549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:59:19.535909 6549 hybrid_clock.cc:648] HybridClock initialized: now 1790225959535892 us; error 35 us; skew 500 ppm
Sep 24 04:59:19 dist-test-slave-70st krb5kdc[6442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225959, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:59:19.538264 6549 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260924 04:59:19.538846 6549 webserver.cc:460] Webserver started at http://127.0.98.1:43981/ using document root <none> and password file <none>
I20260924 04:59:19.539063 6549 fs_manager.cc:359] Metadata directory not provided
I20260924 04:59:19.539112 6549 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:59:19.539244 6549 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:59:19.540269 6549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/ts-0/data/instance:
uuid: "2b2dd144b90544c5bdbefde04c6672d5"
format_stamp: "Formatted at 2026-09-24 04:59:19 on dist-test-slave-70st"
server_key: "77f12540beb6d735f20a79451546b095"
server_key_iv: "c15f5e8e61f6ff60cb7d4aa49f8aaabc"
server_key_version: "encryptionkey@0"
I20260924 04:59:19.540596 6549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/ts-0/wal/instance:
uuid: "2b2dd144b90544c5bdbefde04c6672d5"
format_stamp: "Formatted at 2026-09-24 04:59:19 on dist-test-slave-70st"
server_key: "77f12540beb6d735f20a79451546b095"
server_key_iv: "c15f5e8e61f6ff60cb7d4aa49f8aaabc"
server_key_version: "encryptionkey@0"
I20260924 04:59:19.541811 6549 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260924 04:59:19.542433 6564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:19.542583 6549 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 04:59:19.542667 6549 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/ts-0/wal
uuid: "2b2dd144b90544c5bdbefde04c6672d5"
format_stamp: "Formatted at 2026-09-24 04:59:19 on dist-test-slave-70st"
server_key: "77f12540beb6d735f20a79451546b095"
server_key_iv: "c15f5e8e61f6ff60cb7d4aa49f8aaabc"
server_key_version: "encryptionkey@0"
I20260924 04:59:19.542734 6549 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:59:19.566449 6549 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:59:19.568235 6549 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:59:19.568375 6549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:59:19.568593 6549 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:59:19.568897 6549 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:59:19.568933 6549 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:19.568964 6549 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:59:19.568986 6549 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:19.574697 6549 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46631
I20260924 04:59:19.574752 6677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46631 every 8 connection(s)
I20260924 04:59:19.575078 6549 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 04:59:19 dist-test-slave-70st krb5kdc[6442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225959, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 04:59:19.583182 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 6549
I20260924 04:59:19.583304 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:59:19.583515 392 external_mini_cluster.cc:1428] Setting key 5ddb0f6a949cfd1fd820536f3f6c9abf
I20260924 04:59:19.583855 6527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:19.575749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42055 (local address 127.0.98.62:43351)
0924 04:59:19.575806 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:19.575808 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:19.576026 (+ 218us) server_negotiation.cc:407] Connection header received
0924 04:59:19.576539 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:19.576541 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:19.576552 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:19.576603 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:19.577524 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:19.578050 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:19.579029 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:19.579245 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:19.579320 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:19.581313 (+ 1993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:19.581314 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:19.581315 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:59:19.581321 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:19.582562 (+ 1241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:19.582892 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:19.582892 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:19.582893 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:19.582941 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:19.583118 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:19.583118 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:19.583119 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:19.583250 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:19.583315 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:19.583664 (+ 349us) server_negotiation.cc:299] Negotiation successful
0924 04:59:19.583755 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 04:59:19.584046 6680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:19.575748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43351 (local address 127.0.98.1:42055)
0924 04:59:19.575904 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:19.575941 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:19.576436 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:19.576673 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:19.576676 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:19.577229 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:19.577440 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:19.577445 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:19.578125 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:19.578125 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:19.578931 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:19.578933 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:19.579332 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:19.579332 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:19.579448 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:19.579938 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:19.579946 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:19.581226 (+ 1280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:59:19.582625 (+ 1399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:19.582631 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:19.582632 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:19.582831 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:19.582983 (+ 152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:19.582984 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:19.582984 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:19.583070 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:19.583347 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:59:19.583354 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:59:19.583526 (+ 172us) client_negotiation.cc:769] Sending connection context
0924 04:59:19.583644 (+ 118us) client_negotiation.cc:240] Negotiation successful
0924 04:59:19.583801 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":67,"threads_started":1}
I20260924 04:59:19.584582 6678 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43351
I20260924 04:59:19.584673 6678 heartbeater.cc:464] Registering TS with master...
I20260924 04:59:19.584838 6678 heartbeater.cc:511] Master 127.0.98.62:43351 requested a full tablet report, sending...
I20260924 04:59:19.585395 6487 ts_manager.cc:194] Registered new tserver with Master: 2b2dd144b90544c5bdbefde04c6672d5 (127.0.98.1:46631)
I20260924 04:59:19.586030 6487 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:42055
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260924 04:59:19.608139 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43351
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:59:19.686017 6683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:59:19.686205 6683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:59:19.686225 6683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:59:19.687275 6683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:59:19.687314 6683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:59:19.687337 6683 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:59:19.688683 6683 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43351
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6683
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 04:59:19.688855 6683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:59:19.689019 6683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:59:19.693596 6689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:59:19.693775 6683 server_base.cc:1034] running on GCE node
W20260924 04:59:19.693604 6691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:59:19.693604 6688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:59:19.694064 6683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:59:19.694301 6683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:59:19.695465 6683 hybrid_clock.cc:648] HybridClock initialized: now 1790225959695451 us; error 35 us; skew 500 ppm
Sep 24 04:59:19 dist-test-slave-70st krb5kdc[6442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225959, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:59:19.697945 6683 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260924 04:59:19.698331 6683 webserver.cc:460] Webserver started at http://127.0.98.2:38481/ using document root <none> and password file <none>
I20260924 04:59:19.698504 6683 fs_manager.cc:359] Metadata directory not provided
I20260924 04:59:19.698542 6683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:59:19.698629 6683 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:59:19.699609 6683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/ts-1/data/instance:
uuid: "fc45fc89660947038800b5a9922ea04c"
format_stamp: "Formatted at 2026-09-24 04:59:19 on dist-test-slave-70st"
server_key: "4d10af10afb7282764a00aa1b31791cd"
server_key_iv: "c7d57dbe70cb6e64f34416d19ef4fefd"
server_key_version: "encryptionkey@0"
I20260924 04:59:19.699896 6683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/ts-1/wal/instance:
uuid: "fc45fc89660947038800b5a9922ea04c"
format_stamp: "Formatted at 2026-09-24 04:59:19 on dist-test-slave-70st"
server_key: "4d10af10afb7282764a00aa1b31791cd"
server_key_iv: "c7d57dbe70cb6e64f34416d19ef4fefd"
server_key_version: "encryptionkey@0"
I20260924 04:59:19.701081 6683 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 04:59:19.701776 6698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:19.701925 6683 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:19.702000 6683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/ts-1/wal
uuid: "fc45fc89660947038800b5a9922ea04c"
format_stamp: "Formatted at 2026-09-24 04:59:19 on dist-test-slave-70st"
server_key: "4d10af10afb7282764a00aa1b31791cd"
server_key_iv: "c7d57dbe70cb6e64f34416d19ef4fefd"
server_key_version: "encryptionkey@0"
I20260924 04:59:19.702070 6683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:59:19.722512 6683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:59:19.724431 6683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:59:19.724565 6683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:59:19.724807 6683 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:59:19.725113 6683 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:59:19.725150 6683 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:19.725178 6683 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:59:19.725204 6683 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:19.731101 6683 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42241
I20260924 04:59:19.731153 6811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42241 every 8 connection(s)
I20260924 04:59:19.731459 6683 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:59:19.732739 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 6683
I20260924 04:59:19.732831 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:59:19.733012 392 external_mini_cluster.cc:1428] Setting key 673a853a859d020d4e8a208b993dbbe7
Sep 24 04:59:19 dist-test-slave-70st krb5kdc[6442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225959, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 04:59:19.741075 6527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:19.732097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55673 (local address 127.0.98.62:43351)
0924 04:59:19.732138 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:19.732140 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:19.732327 (+ 187us) server_negotiation.cc:407] Connection header received
0924 04:59:19.732946 (+ 619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:19.732947 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:19.732960 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:19.733019 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:19.734202 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:19.734747 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:19.735820 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:19.736106 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:19.736185 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:19.738207 (+ 2022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:19.738209 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:19.738209 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:59:19.738225 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:19.739730 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:19.740106 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:19.740107 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:19.740107 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:19.740174 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:19.740428 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:19.740429 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:19.740429 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:19.740593 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:19.740675 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:19.740898 (+ 223us) server_negotiation.cc:299] Negotiation successful
0924 04:59:19.740979 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 04:59:19.741187 6814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:19.732076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43351 (local address 127.0.98.2:55673)
0924 04:59:19.732214 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:19.732250 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:19.732820 (+ 570us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:19.733095 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:19.733099 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:19.733806 (+ 707us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:19.734101 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:19.734108 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:19.734837 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:19.734837 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:19.735737 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:19.735739 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:19.736186 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:19.736186 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:19.736287 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:19.736767 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:19.736778 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:19.738117 (+ 1339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:59:19.739818 (+ 1701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:19.739824 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:19.739825 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:19.740036 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:19.740263 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:19.740263 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:19.740264 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:19.740347 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:19.740661 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:59:19.740664 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:59:19.740780 (+ 116us) client_negotiation.cc:769] Sending connection context
0924 04:59:19.740857 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 04:59:19.740983 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":72,"threads_started":1}
I20260924 04:59:19.741628 6812 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43351
I20260924 04:59:19.741724 6812 heartbeater.cc:464] Registering TS with master...
I20260924 04:59:19.741904 6812 heartbeater.cc:511] Master 127.0.98.62:43351 requested a full tablet report, sending...
I20260924 04:59:19.742270 6487 ts_manager.cc:194] Registered new tserver with Master: fc45fc89660947038800b5a9922ea04c (127.0.98.2:42241)
I20260924 04:59:19.742625 6487 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55673
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260924 04:59:19.760725 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43351
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:59:19.839699 6817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:59:19.839908 6817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:59:19.839931 6817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:59:19.841012 6817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:59:19.841051 6817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:59:19.841076 6817 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:59:19.842468 6817 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43351
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6817
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 04:59:19.842677 6817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:59:19.842864 6817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:59:19.847566 6823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:59:19.847564 6825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:59:19.847775 6817 server_base.cc:1034] running on GCE node
W20260924 04:59:19.847595 6822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:59:19.848012 6817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:59:19.848215 6817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:59:19.849355 6817 hybrid_clock.cc:648] HybridClock initialized: now 1790225959849339 us; error 28 us; skew 500 ppm
Sep 24 04:59:19 dist-test-slave-70st krb5kdc[6442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225959, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:59:19.851656 6817 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260924 04:59:19.852087 6817 webserver.cc:460] Webserver started at http://127.0.98.3:37593/ using document root <none> and password file <none>
I20260924 04:59:19.852273 6817 fs_manager.cc:359] Metadata directory not provided
I20260924 04:59:19.852316 6817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:59:19.852437 6817 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:59:19.853531 6817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/ts-2/data/instance:
uuid: "b08a35b7530a47f48a8a4cf653551a6a"
format_stamp: "Formatted at 2026-09-24 04:59:19 on dist-test-slave-70st"
server_key: "971d9f66deaadd968a5419da172d60e6"
server_key_iv: "1a109846bbbe47e03c431940d2cfbe08"
server_key_version: "encryptionkey@0"
I20260924 04:59:19.853868 6817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/ts-2/wal/instance:
uuid: "b08a35b7530a47f48a8a4cf653551a6a"
format_stamp: "Formatted at 2026-09-24 04:59:19 on dist-test-slave-70st"
server_key: "971d9f66deaadd968a5419da172d60e6"
server_key_iv: "1a109846bbbe47e03c431940d2cfbe08"
server_key_version: "encryptionkey@0"
I20260924 04:59:19.855018 6817 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 04:59:19.855616 6832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:19.855762 6817 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 04:59:19.855842 6817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/ts-2/wal
uuid: "b08a35b7530a47f48a8a4cf653551a6a"
format_stamp: "Formatted at 2026-09-24 04:59:19 on dist-test-slave-70st"
server_key: "971d9f66deaadd968a5419da172d60e6"
server_key_iv: "1a109846bbbe47e03c431940d2cfbe08"
server_key_version: "encryptionkey@0"
I20260924 04:59:19.855897 6817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:59:19.868772 6817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:59:19.870504 6817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:59:19.870606 6817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:59:19.870784 6817 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:59:19.871024 6817 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:59:19.871052 6817 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:19.871073 6817 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:59:19.871088 6817 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:19.876389 6817 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44049
I20260924 04:59:19.876446 6945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44049 every 8 connection(s)
I20260924 04:59:19.876709 6817 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 04:59:19 dist-test-slave-70st krb5kdc[6442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225959, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 04:59:19.885177 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 6817
I20260924 04:59:19.885200 6527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:19.877446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51567 (local address 127.0.98.62:43351)
0924 04:59:19.877491 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:19.877493 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:19.877660 (+ 167us) server_negotiation.cc:407] Connection header received
0924 04:59:19.878145 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:19.878147 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:19.878161 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:19.878212 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:19.879116 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:19.879603 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:19.880562 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:19.880746 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:19.880806 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:19.882745 (+ 1939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:19.882747 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:19.882747 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:59:19.882755 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:19.883989 (+ 1234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:19.884343 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:19.884344 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:19.884344 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:19.884397 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:19.884604 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:19.884605 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:19.884605 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:19.884714 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:19.884770 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:19.885033 (+ 263us) server_negotiation.cc:299] Negotiation successful
0924 04:59:19.885111 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 04:59:19.885272 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:59:19.885341 6948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:19.877378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43351 (local address 127.0.98.3:51567)
0924 04:59:19.877546 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:19.877581 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:19.878047 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:19.878279 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:19.878282 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:19.878818 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:19.879018 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:19.879023 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:19.879681 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:19.879682 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:19.880477 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:19.880480 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:19.880816 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:19.880817 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:19.880930 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:19.881353 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:19.881361 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:19.882660 (+ 1299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:59:19.884055 (+ 1395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:19.884060 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:19.884062 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:19.884276 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:19.884448 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:19.884449 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:19.884449 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:19.884550 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:19.884765 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:59:19.884770 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:59:19.884903 (+ 133us) client_negotiation.cc:769] Sending connection context
0924 04:59:19.884986 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 04:59:19.885133 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":104,"threads_started":1}
I20260924 04:59:19.885511 392 external_mini_cluster.cc:1428] Setting key bd37b54cf480f7bca07e33f03d074acc
I20260924 04:59:19.885820 6946 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43351
I20260924 04:59:19.885926 6946 heartbeater.cc:464] Registering TS with master...
I20260924 04:59:19.886134 6946 heartbeater.cc:511] Master 127.0.98.62:43351 requested a full tablet report, sending...
I20260924 04:59:19.886447 6487 ts_manager.cc:194] Registered new tserver with Master: b08a35b7530a47f48a8a4cf653551a6a (127.0.98.3:44049)
I20260924 04:59:19.886807 6487 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:51567
I20260924 04:59:19.897342 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260924 04:59:19.921973 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:40695
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43351,127.0.98.61:40695
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:59:19.999630 6951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:59:19.999819 6951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:59:19.999838 6951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:59:20.000874 6951 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:59:20.000905 6951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:59:20.000918 6951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:59:20.000929 6951 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:59:20.000939 6951 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:59:20.002372 6951 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43351,127.0.98.61:40695
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:40695
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6951
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
W20260924 04:59:20.002470 6951 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:43351,127.0.98.61:40695'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260924 04:59:20.002574 6951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:59:20.002731 6951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:59:20.007225 6956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:59:20.007308 6951 server_base.cc:1034] running on GCE node
W20260924 04:59:20.007228 6959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:59:20.007289 6957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:59:20.007714 6951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:59:20.007983 6951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:59:20.009121 6951 hybrid_clock.cc:648] HybridClock initialized: now 1790225960009107 us; error 29 us; skew 500 ppm
Sep 24 04:59:20 dist-test-slave-70st krb5kdc[6442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225960, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:59:20.019867 6951 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260924 04:59:20.020350 6951 webserver.cc:460] Webserver started at http://127.0.98.61:46229/ using document root <none> and password file <none>
I20260924 04:59:20.020556 6951 fs_manager.cc:359] Metadata directory not provided
I20260924 04:59:20.020603 6951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:59:20.020726 6951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:59:20.021843 6951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/master-1/data/instance:
uuid: "c82275d62aac4bc580771ab80ac00860"
format_stamp: "Formatted at 2026-09-24 04:59:20 on dist-test-slave-70st"
server_key: "7e315d60334b4ce981d8b04031d8cda0"
server_key_iv: "3d046a51d8b3db863651c39cd7e1c468"
server_key_version: "encryptionkey@0"
I20260924 04:59:20.022172 6951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/master-1/wal/instance:
uuid: "c82275d62aac4bc580771ab80ac00860"
format_stamp: "Formatted at 2026-09-24 04:59:20 on dist-test-slave-70st"
server_key: "7e315d60334b4ce981d8b04031d8cda0"
server_key_iv: "3d046a51d8b3db863651c39cd7e1c468"
server_key_version: "encryptionkey@0"
I20260924 04:59:20.023398 6951 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 04:59:20.024086 6966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:20.024253 6951 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 04:59:20.024335 6951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/master-1/wal
uuid: "c82275d62aac4bc580771ab80ac00860"
format_stamp: "Formatted at 2026-09-24 04:59:20 on dist-test-slave-70st"
server_key: "7e315d60334b4ce981d8b04031d8cda0"
server_key_iv: "3d046a51d8b3db863651c39cd7e1c468"
server_key_version: "encryptionkey@0"
I20260924 04:59:20.024394 6951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:59:20.047897 6951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:59:20.050088 6951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:59:20.050225 6951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:59:20.054132 6951 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:40695
I20260924 04:59:20.054165 7018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:40695 every 8 connection(s)
I20260924 04:59:20.054503 6951 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/master-1/data/info.pb
I20260924 04:59:20.055423 7019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:59:20.056519 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 6951
I20260924 04:59:20.057163 7019 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43351 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 04:59:20 dist-test-slave-70st krb5kdc[6442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225960, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 24 04:59:20 dist-test-slave-70st krb5kdc[6442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225959, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260924 04:59:20.067030 6527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:20.057874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34282 (local address 127.0.98.62:43351)
0924 04:59:20.057926 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:20.057928 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:20.058078 (+ 150us) server_negotiation.cc:407] Connection header received
0924 04:59:20.058129 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:20.058130 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:20.058141 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:20.058216 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:20.058747 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.059553 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.060888 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.061147 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.061223 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:20.063896 (+ 2673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:20.063898 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:20.063898 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:59:20.063908 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:20.065662 (+ 1754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:20.066020 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:20.066021 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:20.066022 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:20.066082 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:20.066339 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:20.066340 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:20.066340 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:20.066496 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:20.066561 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:20.066850 (+ 289us) server_negotiation.cc:299] Negotiation successful
0924 04:59:20.066935 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 04:59:20.067189 7023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:20.057784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43351 (local address 127.0.0.1:34282)
0924 04:59:20.057955 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:20.057997 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:20.058075 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:20.058237 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:20.058239 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:20.058501 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:20.058668 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:20.058672 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.059672 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.059673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:20.060791 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:20.060794 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.061276 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.061277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:20.061378 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:20.061945 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:20.061957 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:20.063797 (+ 1840us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:59:20.065734 (+ 1937us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:20.065740 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:20.065742 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:20.065948 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:20.066150 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:20.066151 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:20.066151 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:20.066277 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:20.066557 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:59:20.066561 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:59:20.066714 (+ 153us) client_negotiation.cc:769] Sending connection context
0924 04:59:20.066818 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 04:59:20.066942 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":100,"threads_started":1}
I20260924 04:59:20.067488 7022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:20.057150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45642 (local address 127.0.98.61:40695)
0924 04:59:20.057331 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:20.057342 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:20.057383 (+ 41us) server_negotiation.cc:407] Connection header received
0924 04:59:20.057896 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:20.057905 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:20.057971 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:20.058124 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:20.058819 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.059937 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.060978 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.061380 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.061461 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:20.064538 (+ 3077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:20.064546 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:20.064547 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:59:20.064554 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:20.066189 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:20.066521 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:20.066524 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:20.066524 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:20.066580 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:20.066763 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:20.066764 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:20.066764 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:20.066907 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:20.067030 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:20.067333 (+ 303us) server_negotiation.cc:299] Negotiation successful
0924 04:59:20.067404 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":87,"threads_started":1}
I20260924 04:59:20.067821 7019 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40695 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 04:59:20 dist-test-slave-70st krb5kdc[6442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225960, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260924 04:59:20.073817 7023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:20.068050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40695 (local address 127.0.0.1:45658)
0924 04:59:20.068083 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:20.068094 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:20.068136 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:20.068231 (+ 95us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:20.068232 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:20.068341 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:20.068388 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:20.068390 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.068975 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.068976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:20.069529 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:20.069531 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.069845 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.069846 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:20.069890 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:20.070239 (+ 349us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:20.070243 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:20.071384 (+ 1141us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:59:20.072859 (+ 1475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:20.072860 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:20.072861 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:20.073029 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:20.073221 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:20.073221 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:20.073222 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:20.073264 (+ 42us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:20.073575 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:59:20.073575 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:59:20.073611 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 04:59:20.073656 (+ 45us) client_negotiation.cc:240] Negotiation successful
0924 04:59:20.073733 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17}
I20260924 04:59:20.073894 7022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:20.068081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45658 (local address 127.0.98.61:40695)
0924 04:59:20.068111 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:20.068112 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:20.068123 (+ 11us) server_negotiation.cc:407] Connection header received
0924 04:59:20.068187 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:20.068187 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:20.068194 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:20.068229 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:20.068437 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.068895 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.069588 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.069790 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.069857 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:20.071509 (+ 1652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:20.071510 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:20.071510 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:59:20.071516 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:20.072785 (+ 1269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:20.073106 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:20.073107 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:20.073107 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:20.073160 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:20.073334 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:20.073335 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:20.073335 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:20.073513 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:20.073573 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:20.073683 (+ 110us) server_negotiation.cc:299] Negotiation successful
0924 04:59:20.073785 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":14}
I20260924 04:59:20.075124 7019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c82275d62aac4bc580771ab80ac00860: Bootstrap starting.
I20260924 04:59:20.075943 7019 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c82275d62aac4bc580771ab80ac00860: Neither blocks nor log segments found. Creating new log.
I20260924 04:59:20.076205 7019 log.cc:824] T 00000000000000000000000000000000 P c82275d62aac4bc580771ab80ac00860: Log is configured to *not* fsync() on all Append() calls
I20260924 04:59:20.076927 7019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c82275d62aac4bc580771ab80ac00860: No bootstrap required, opened a new log
I20260924 04:59:20.078539 7019 raft_consensus.cc:348] T 00000000000000000000000000000000 P c82275d62aac4bc580771ab80ac00860 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "221d34b534b046078e9003aff8289b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43351 } } peers { permanent_uuid: "c82275d62aac4bc580771ab80ac00860" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40695 } }
I20260924 04:59:20.078702 7019 raft_consensus.cc:374] T 00000000000000000000000000000000 P c82275d62aac4bc580771ab80ac00860 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:59:20.078743 7019 raft_consensus.cc:729] T 00000000000000000000000000000000 P c82275d62aac4bc580771ab80ac00860 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c82275d62aac4bc580771ab80ac00860, State: Initialized, Role: FOLLOWER
I20260924 04:59:20.078893 7019 consensus_queue.cc:260] T 00000000000000000000000000000000 P c82275d62aac4bc580771ab80ac00860 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "221d34b534b046078e9003aff8289b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43351 } } peers { permanent_uuid: "c82275d62aac4bc580771ab80ac00860" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40695 } }
I20260924 04:59:20.079151 7025 sys_catalog.cc:455] T 00000000000000000000000000000000 P c82275d62aac4bc580771ab80ac00860 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "221d34b534b046078e9003aff8289b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43351 } } peers { permanent_uuid: "c82275d62aac4bc580771ab80ac00860" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40695 } } }
I20260924 04:59:20.079227 7019 sys_catalog.cc:564] T 00000000000000000000000000000000 P c82275d62aac4bc580771ab80ac00860 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:59:20.079237 7025 sys_catalog.cc:458] T 00000000000000000000000000000000 P c82275d62aac4bc580771ab80ac00860 [sys.catalog]: This master's current role is: FOLLOWER
W20260924 04:59:20.081176 7036 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c82275d62aac4bc580771ab80ac00860: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:59:20.081277 7036 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:59:20.082855 6951 master_runner.cc:428] Detected that this master c82275d62aac4bc580771ab80ac00860 is joining an existing cluster
I20260924 04:59:20.082911 6951 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:40695
I20260924 04:59:20.087952 7025 raft_consensus.cc:482] T 00000000000000000000000000000000 P c82275d62aac4bc580771ab80ac00860 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:59:20.088047 7025 raft_consensus.cc:504] T 00000000000000000000000000000000 P c82275d62aac4bc580771ab80ac00860 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "221d34b534b046078e9003aff8289b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43351 } } peers { permanent_uuid: "c82275d62aac4bc580771ab80ac00860" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40695 } }
I20260924 04:59:20.088276 7025 leader_election.cc:288] T 00000000000000000000000000000000 P c82275d62aac4bc580771ab80ac00860 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 221d34b534b046078e9003aff8289b32 (127.0.98.62:43351)
I20260924 04:59:20.088527 6497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c82275d62aac4bc580771ab80ac00860" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "221d34b534b046078e9003aff8289b32" is_pre_election: true
I20260924 04:59:20.088888 6970 leader_election.cc:302] T 00000000000000000000000000000000 P c82275d62aac4bc580771ab80ac00860 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: c82275d62aac4bc580771ab80ac00860; no voters: 221d34b534b046078e9003aff8289b32
I20260924 04:59:20.089035 7025 raft_consensus.cc:2726] T 00000000000000000000000000000000 P c82275d62aac4bc580771ab80ac00860 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 04:59:20.091329 7043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:20.085138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40695 (local address 127.0.0.1:45660)
0924 04:59:20.085258 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:20.085267 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:20.085320 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:20.085476 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:20.085477 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:20.085659 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:20.085711 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:20.085713 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.086536 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.086537 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:20.087395 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:20.087396 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.087684 (+ 288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.087685 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:20.087736 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:20.088423 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:20.088427 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:20.088849 (+ 422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:59:20.090399 (+ 1550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:20.090400 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:20.090401 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:20.090600 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:20.090757 (+ 157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:20.090757 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:20.090758 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:20.090809 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:20.091022 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:59:20.091022 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:59:20.091057 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 04:59:20.091115 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 04:59:20.091213 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":52,"threads_started":1}
I20260924 04:59:20.091336 7022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:20.085178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45660 (local address 127.0.98.61:40695)
0924 04:59:20.085214 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:20.085215 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:20.085319 (+ 104us) server_negotiation.cc:407] Connection header received
0924 04:59:20.085377 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:20.085378 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:20.085386 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:20.085446 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:20.085779 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.086392 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.087451 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.087632 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.087682 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:20.088932 (+ 1250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:20.088933 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:20.088933 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:59:20.088939 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:20.090323 (+ 1384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:20.090659 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:20.090660 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:20.090660 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:20.090708 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:20.090856 (+ 148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:20.090856 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:20.090857 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:20.090973 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:20.091024 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:20.091139 (+ 115us) server_negotiation.cc:299] Negotiation successful
0924 04:59:20.091258 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":80,"server-negotiator.queue_time_us":19}
I20260924 04:59:20.092000 6527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:20.083919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34294 (local address 127.0.98.62:43351)
0924 04:59:20.083962 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:20.083965 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:20.084118 (+ 153us) server_negotiation.cc:407] Connection header received
0924 04:59:20.084184 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:20.084186 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:20.084195 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:20.084240 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:20.084826 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.085616 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.087330 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.087637 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.087707 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:20.089077 (+ 1370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:20.089079 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:20.089079 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:59:20.089086 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:20.090629 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:20.091023 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:20.091025 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:20.091025 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:20.091087 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:20.091269 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:20.091270 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:20.091270 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:20.091386 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:20.091431 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:20.091856 (+ 425us) server_negotiation.cc:299] Negotiation successful
0924 04:59:20.091940 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 04:59:20.092211 7042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:20.083939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43351 (local address 127.0.0.1:34294)
0924 04:59:20.084065 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:20.084074 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:20.084128 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:20.084281 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:20.084282 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:20.084453 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:20.084753 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:20.084755 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.085695 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.085696 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:20.087242 (+ 1546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:20.087244 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.087763 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.087765 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:20.087834 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:20.088309 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:20.088352 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:20.088901 (+ 549us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:59:20.090724 (+ 1823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:20.090725 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:20.090726 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:20.090935 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:20.091154 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:20.091154 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:20.091155 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:20.091209 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:20.091707 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:59:20.091708 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:59:20.091751 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 04:59:20.091814 (+ 63us) client_negotiation.cc:240] Negotiation successful
0924 04:59:20.091918 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":79,"threads_started":1}
Sep 24 04:59:20 dist-test-slave-70st krb5kdc[6442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225959, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260924 04:59:20.101186 7022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:20.094128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45668 (local address 127.0.98.61:40695)
0924 04:59:20.094159 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:20.094161 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:20.094325 (+ 164us) server_negotiation.cc:407] Connection header received
0924 04:59:20.094408 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:20.094408 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:20.094415 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:20.094457 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:20.094908 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.095344 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.096429 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.096597 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.096645 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:20.098741 (+ 2096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:20.098742 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:20.098743 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:59:20.098749 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:20.100013 (+ 1264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:20.100456 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:20.100457 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:20.100458 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:20.100527 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:20.100709 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:20.100710 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:20.100710 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:20.100810 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:20.100856 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:20.101056 (+ 200us) server_negotiation.cc:299] Negotiation successful
0924 04:59:20.101120 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260924 04:59:20.101270 7044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:20.094081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40695 (local address 127.0.0.1:45668)
0924 04:59:20.094233 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:20.094273 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:20.094339 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:20.094463 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:20.094466 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:20.094783 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:20.094840 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:20.094842 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.095406 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.095407 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:20.096352 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:20.096355 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.096657 (+ 302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.096658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:20.096744 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:20.097133 (+ 389us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:20.097138 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:20.098643 (+ 1505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:59:20.100086 (+ 1443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:20.100095 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:20.100097 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:20.100370 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:20.100593 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:20.100594 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:20.100594 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:20.100657 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:20.100859 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:59:20.100888 (+ 29us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:59:20.100947 (+ 59us) client_negotiation.cc:769] Sending connection context
0924 04:59:20.101029 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 04:59:20.101151 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":54,"threads_started":1}
I20260924 04:59:20.101877 6487 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:40695: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "c82275d62aac4bc580771ab80ac00860"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 40695
}
attrs {
promote: true
}
}
dest_uuid: "221d34b534b046078e9003aff8289b32"
cas_config_opid_index: -1
I20260924 04:59:20.102190 6487 consensus_queue.cc:237] T 00000000000000000000000000000000 P 221d34b534b046078e9003aff8289b32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "221d34b534b046078e9003aff8289b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43351 } } peers { permanent_uuid: "c82275d62aac4bc580771ab80ac00860" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40695 } attrs { promote: true } }
I20260924 04:59:20.102649 7046 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 221d34b534b046078e9003aff8289b32 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER c82275d62aac4bc580771ab80ac00860 (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "221d34b534b046078e9003aff8289b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43351 } } peers { permanent_uuid: "c82275d62aac4bc580771ab80ac00860" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40695 } attrs { promote: true } } }
I20260924 04:59:20.102928 6993 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c82275d62aac4bc580771ab80ac00860 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:59:20.103504 7046 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:40695
I20260924 04:59:20.103753 6951 master.cc:537] Master@127.0.98.61:40695 shutting down...
I20260924 04:59:20.103806 6993 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c82275d62aac4bc580771ab80ac00860 [term 1 FOLLOWER]: Refusing update from remote peer 221d34b534b046078e9003aff8289b32: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260924 04:59:20.104094 7046 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 221d34b534b046078e9003aff8289b32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c82275d62aac4bc580771ab80ac00860" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40695 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:59:20.103753 7045 sys_catalog.cc:455] T 00000000000000000000000000000000 P 221d34b534b046078e9003aff8289b32 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "221d34b534b046078e9003aff8289b32" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "221d34b534b046078e9003aff8289b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43351 } } peers { permanent_uuid: "c82275d62aac4bc580771ab80ac00860" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40695 } attrs { promote: true } } }
W20260924 04:59:20.104772 6471 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260924 04:59:20.104946 7045 sys_catalog.cc:458] T 00000000000000000000000000000000 P 221d34b534b046078e9003aff8289b32 [sys.catalog]: This master's current role is: LEADER
W20260924 04:59:20.105185 6471 consensus_peers.cc:487] T 00000000000000000000000000000000 P 221d34b534b046078e9003aff8289b32 -> Peer c82275d62aac4bc580771ab80ac00860 (127.0.98.61:40695): Couldn't send request to peer c82275d62aac4bc580771ab80ac00860. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260924 04:59:20.106698 6951 raft_consensus.cc:2229] T 00000000000000000000000000000000 P c82275d62aac4bc580771ab80ac00860 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:59:20.106777 6951 raft_consensus.cc:2258] T 00000000000000000000000000000000 P c82275d62aac4bc580771ab80ac00860 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:59:20.106820 6951 tablet_replica.cc:330] T 00000000000000000000000000000000 P c82275d62aac4bc580771ab80ac00860: stopping tablet replica
I20260924 04:59:20.119270 6951 master.cc:552] Master@127.0.98.61:40695 shutdown complete.
I20260924 04:59:20.119330 6951 master_runner.cc:305] Clearing existing system tablet
I20260924 04:59:20.119441 6951 fs_manager.cc:359] Metadata directory not provided
I20260924 04:59:20.119483 6951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:59:20.120057 6951 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260924 04:59:20.120663 7050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:20.120790 6951 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:59:20.120841 6951 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 04:59:20.120862 6951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/master-1/wal
uuid: "c82275d62aac4bc580771ab80ac00860"
format_stamp: "Formatted at 2026-09-24 04:59:20 on dist-test-slave-70st"
server_key: "7e315d60334b4ce981d8b04031d8cda0"
server_key_iv: "3d046a51d8b3db863651c39cd7e1c468"
server_key_version: "encryptionkey@0"
I20260924 04:59:20.121592 6951 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P c82275d62aac4bc580771ab80ac00860: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:59:20.122992 6951 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P c82275d62aac4bc580771ab80ac00860: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 04:59:20.123057 6951 log.cc:1196] T 00000000000000000000000000000000 P c82275d62aac4bc580771ab80ac00860: Deleting WAL directory at /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260924 04:59:20.123219 6951 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P c82275d62aac4bc580771ab80ac00860: Deleting consensus metadata
I20260924 04:59:20.123319 6951 master_runner.cc:315] Copying system tablet from 127.0.98.62:43351
I20260924 04:59:20.123874 6951 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P c82275d62aac4bc580771ab80ac00860: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:43351
I20260924 04:59:20.129745 7056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:20.124160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43351 (local address 127.0.0.1:34304)
0924 04:59:20.124262 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:20.124273 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:20.124332 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:20.124445 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:20.124445 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:20.124639 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:20.124703 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:20.124705 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.125357 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.125358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:20.125975 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:20.125977 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.126318 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.126319 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:20.126377 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:20.126905 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:20.126910 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:20.127236 (+ 326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:59:20.128742 (+ 1506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:20.128743 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:20.128744 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:20.128917 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:20.129086 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:20.129086 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:20.129086 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:20.129142 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:20.129359 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:59:20.129361 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:59:20.129456 (+ 95us) client_negotiation.cc:769] Sending connection context
0924 04:59:20.129550 (+ 94us) client_negotiation.cc:240] Negotiation successful
0924 04:59:20.129654 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81,"thread_start_us":64,"threads_started":1}
I20260924 04:59:20.129810 6527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:20.124243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34304 (local address 127.0.98.62:43351)
0924 04:59:20.124302 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:20.124304 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:20.124317 (+ 13us) server_negotiation.cc:407] Connection header received
0924 04:59:20.124385 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:20.124386 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:20.124397 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:20.124442 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:20.124785 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.125282 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.126051 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.126240 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.126317 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:20.127336 (+ 1019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:20.127337 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:20.127337 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:59:20.127344 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:20.128666 (+ 1322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:20.128979 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:20.128980 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:20.128980 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:20.129035 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:20.129190 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:20.129191 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:20.129191 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:20.129306 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:20.129362 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:20.129615 (+ 253us) server_negotiation.cc:299] Negotiation successful
0924 04:59:20.129711 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 04:59:20.130105 6507 tablet_copy_service.cc:140] P 221d34b534b046078e9003aff8289b32: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer c82275d62aac4bc580771ab80ac00860 ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:34304)
I20260924 04:59:20.130192 6507 tablet_copy_service.cc:161] P 221d34b534b046078e9003aff8289b32: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer c82275d62aac4bc580771ab80ac00860 at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:34304: session id = c82275d62aac4bc580771ab80ac00860-00000000000000000000000000000000
I20260924 04:59:20.130811 6507 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 221d34b534b046078e9003aff8289b32: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 04:59:20.131428 6951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 04:59:20.132328 6951 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P c82275d62aac4bc580771ab80ac00860: tablet copy: Starting download of 0 data blocks...
I20260924 04:59:20.132457 6951 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P c82275d62aac4bc580771ab80ac00860: tablet copy: Starting download of 1 WAL segments...
I20260924 04:59:20.133216 6951 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P c82275d62aac4bc580771ab80ac00860: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 04:59:20.134034 6507 tablet_copy_service.cc:342] P 221d34b534b046078e9003aff8289b32: Request end of tablet copy session c82275d62aac4bc580771ab80ac00860-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:34304
I20260924 04:59:20.134083 6507 tablet_copy_service.cc:433] P 221d34b534b046078e9003aff8289b32: ending tablet copy session c82275d62aac4bc580771ab80ac00860-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer c82275d62aac4bc580771ab80ac00860
I20260924 04:59:20.136401 6951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:59:20.137688 7057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:59:20.137717 7060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:59:20.137763 6951 server_base.cc:1034] running on GCE node
W20260924 04:59:20.137830 7058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:59:20.138015 6951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:59:20.138137 6951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:59:20.139277 6951 hybrid_clock.cc:648] HybridClock initialized: now 1790225960139269 us; error 25 us; skew 500 ppm
Sep 24 04:59:20 dist-test-slave-70st krb5kdc[6442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225960, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:59:20.140511 6951 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260924 04:59:20.140734 6951 webserver.cc:460] Webserver started at http://127.0.98.61:40033/ using document root <none> and password file <none>
I20260924 04:59:20.140852 6951 fs_manager.cc:359] Metadata directory not provided
I20260924 04:59:20.140885 6951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:59:20.141484 6951 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:20.141956 7067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:20.142071 6951 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:20.142127 6951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/master-1/wal
uuid: "c82275d62aac4bc580771ab80ac00860"
format_stamp: "Formatted at 2026-09-24 04:59:20 on dist-test-slave-70st"
server_key: "7e315d60334b4ce981d8b04031d8cda0"
server_key_iv: "3d046a51d8b3db863651c39cd7e1c468"
server_key_version: "encryptionkey@0"
I20260924 04:59:20.142182 6951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:59:20.149401 6951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:59:20.151033 6951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:59:20.153990 6951 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:40695
I20260924 04:59:20.154378 6951 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/master-1/data/info.pb
I20260924 04:59:20.154397 7119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:40695 every 8 connection(s)
I20260924 04:59:20.155088 7120 sys_catalog.cc:263] Verifying existing consensus state
I20260924 04:59:20.155452 7120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c82275d62aac4bc580771ab80ac00860: Bootstrap starting.
I20260924 04:59:20.157660 7120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c82275d62aac4bc580771ab80ac00860: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 04:59:20.157847 7120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c82275d62aac4bc580771ab80ac00860: Bootstrap complete.
I20260924 04:59:20.158041 7120 raft_consensus.cc:348] T 00000000000000000000000000000000 P c82275d62aac4bc580771ab80ac00860 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "221d34b534b046078e9003aff8289b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43351 } } peers { permanent_uuid: "c82275d62aac4bc580771ab80ac00860" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40695 } attrs { promote: true } }
I20260924 04:59:20.158116 7120 raft_consensus.cc:729] T 00000000000000000000000000000000 P c82275d62aac4bc580771ab80ac00860 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c82275d62aac4bc580771ab80ac00860, State: Initialized, Role: LEARNER
I20260924 04:59:20.158151 7120 consensus_queue.cc:260] T 00000000000000000000000000000000 P c82275d62aac4bc580771ab80ac00860 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "221d34b534b046078e9003aff8289b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43351 } } peers { permanent_uuid: "c82275d62aac4bc580771ab80ac00860" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40695 } attrs { promote: true } }
I20260924 04:59:20.158313 7120 sys_catalog.cc:564] T 00000000000000000000000000000000 P c82275d62aac4bc580771ab80ac00860 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:59:20.158304 7123 sys_catalog.cc:455] T 00000000000000000000000000000000 P c82275d62aac4bc580771ab80ac00860 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "221d34b534b046078e9003aff8289b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43351 } } peers { permanent_uuid: "c82275d62aac4bc580771ab80ac00860" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40695 } attrs { promote: true } } }
I20260924 04:59:20.158434 7123 sys_catalog.cc:458] T 00000000000000000000000000000000 P c82275d62aac4bc580771ab80ac00860 [sys.catalog]: This master's current role is: LEARNER
I20260924 04:59:20.159615 7134 catalog_manager.cc:1259] Loaded cluster ID: c56b85dc80bd450c8c3816861b8b9cb0
I20260924 04:59:20.159653 7134 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c82275d62aac4bc580771ab80ac00860: loading cluster ID for follower catalog manager: success
I20260924 04:59:20.160831 7134 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c82275d62aac4bc580771ab80ac00860: acquiring CA information for follower catalog manager: success
I20260924 04:59:20.161126 7134 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c82275d62aac4bc580771ab80ac00860: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 04:59:20.176723 7135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:20.169172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45692 (local address 127.0.98.61:40695)
0924 04:59:20.169331 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:20.169333 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:20.169345 (+ 12us) server_negotiation.cc:407] Connection header received
0924 04:59:20.169371 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:20.169371 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:20.169384 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:20.169485 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:20.170032 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.171046 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.171783 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.172105 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.172204 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:20.173948 (+ 1744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:20.173950 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:20.173950 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:59:20.173959 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:20.175545 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:20.175883 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:20.175885 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:20.175885 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:20.175952 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:20.176140 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:20.176141 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:20.176141 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:20.176281 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:20.176384 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:20.176518 (+ 134us) server_negotiation.cc:299] Negotiation successful
0924 04:59:20.176605 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":79,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260924 04:59:20.204595 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:44097
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43351,127.0.98.61:40695,127.0.98.60:44097
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:59:20.294430 7138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:59:20.294629 7138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:59:20.294651 7138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:59:20.295809 7138 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:59:20.295851 7138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:59:20.295871 7138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:59:20.295893 7138 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:59:20.295917 7138 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:59:20.297379 7138 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43351,127.0.98.61:40695,127.0.98.60:44097
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:44097
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7138
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 04:59:20.297640 7138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:59:20.297870 7138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:59:20.302202 7143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:59:20.302227 7146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:59:20.302214 7144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:59:20.302578 7138 server_base.cc:1034] running on GCE node
I20260924 04:59:20.302809 7138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:59:20.303085 7138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:59:20.304216 7138 hybrid_clock.cc:648] HybridClock initialized: now 1790225960304200 us; error 29 us; skew 500 ppm
Sep 24 04:59:20 dist-test-slave-70st krb5kdc[6442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225960, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:59:20.306696 7138 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260924 04:59:20.307077 7138 webserver.cc:460] Webserver started at http://127.0.98.60:43709/ using document root <none> and password file <none>
I20260924 04:59:20.307237 7138 fs_manager.cc:359] Metadata directory not provided
I20260924 04:59:20.307274 7138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:59:20.307361 7138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:59:20.308329 7138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/master-2/data/instance:
uuid: "a65aed7f2fdc4e6ea4e8765bbe365a42"
format_stamp: "Formatted at 2026-09-24 04:59:20 on dist-test-slave-70st"
server_key: "5173dee5f88860e4d6b8973925341859"
server_key_iv: "189e7ab7c894a27ae90eb5bf4d07c108"
server_key_version: "encryptionkey@0"
I20260924 04:59:20.308614 7138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/master-2/wal/instance:
uuid: "a65aed7f2fdc4e6ea4e8765bbe365a42"
format_stamp: "Formatted at 2026-09-24 04:59:20 on dist-test-slave-70st"
server_key: "5173dee5f88860e4d6b8973925341859"
server_key_iv: "189e7ab7c894a27ae90eb5bf4d07c108"
server_key_version: "encryptionkey@0"
I20260924 04:59:20.309820 7138 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 04:59:20.310474 7153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:20.310680 7138 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 04:59:20.310766 7138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/master-2/wal
uuid: "a65aed7f2fdc4e6ea4e8765bbe365a42"
format_stamp: "Formatted at 2026-09-24 04:59:20 on dist-test-slave-70st"
server_key: "5173dee5f88860e4d6b8973925341859"
server_key_iv: "189e7ab7c894a27ae90eb5bf4d07c108"
server_key_version: "encryptionkey@0"
I20260924 04:59:20.310822 7138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:59:20.320554 7138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:59:20.322193 7138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:59:20.322296 7138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:59:20.326390 7138 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:44097
I20260924 04:59:20.326435 7205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:44097 every 8 connection(s)
I20260924 04:59:20.326735 7138 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/master-2/data/info.pb
I20260924 04:59:20.327587 7206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:59:20.328998 7206 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43351 } has no permanent_uuid. Determining permanent_uuid...
I20260924 04:59:20.329584 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 7138
Sep 24 04:59:20 dist-test-slave-70st krb5kdc[6442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225960, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 24 04:59:20 dist-test-slave-70st krb5kdc[6442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225959, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260924 04:59:20.338570 6527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:20.329829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34318 (local address 127.0.98.62:43351)
0924 04:59:20.329877 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:20.329879 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:20.330094 (+ 215us) server_negotiation.cc:407] Connection header received
0924 04:59:20.330658 (+ 564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:20.330660 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:20.330672 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:20.330741 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:20.331248 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.331785 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.332782 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.333074 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.333122 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:20.335571 (+ 2449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:20.335573 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:20.335574 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:59:20.335581 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:20.337016 (+ 1435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:20.337452 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:20.337453 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:20.337453 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:20.337510 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:20.337965 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:20.337966 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:20.337966 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:20.338117 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:20.338175 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:20.338406 (+ 231us) server_negotiation.cc:299] Negotiation successful
0924 04:59:20.338479 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 04:59:20.338783 7208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:20.329682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43351 (local address 127.0.0.1:34318)
0924 04:59:20.329959 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:20.330006 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:20.330521 (+ 515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:20.330772 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:20.330775 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:20.331018 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:20.331163 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:20.331167 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.331869 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.331870 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:20.332699 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:20.332701 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.333154 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.333154 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:20.333308 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:20.333863 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:20.333871 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:20.335478 (+ 1607us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:59:20.337105 (+ 1627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:20.337111 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:20.337112 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:20.337365 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:20.337802 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:20.337803 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:20.337803 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:20.337896 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:20.338165 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:59:20.338171 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:59:20.338297 (+ 126us) client_negotiation.cc:769] Sending connection context
0924 04:59:20.338376 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 04:59:20.338518 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"mutex_wait_us":24,"thread_start_us":70,"threads_started":1}
I20260924 04:59:20.339185 7209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:20.330045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47242 (local address 127.0.98.60:44097)
0924 04:59:20.330166 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:20.330171 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:20.330197 (+ 26us) server_negotiation.cc:407] Connection header received
0924 04:59:20.330533 (+ 336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:20.330542 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:20.330567 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:20.330701 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:20.331258 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.332253 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.333013 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.333445 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.333517 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:20.335978 (+ 2461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:20.335985 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:20.335985 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:59:20.335992 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:20.337520 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:20.337812 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:20.337815 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:20.337816 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:20.337901 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:20.338118 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:20.338119 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:20.338119 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:20.338323 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:20.338993 (+ 670us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:20.339039 (+ 46us) server_negotiation.cc:299] Negotiation successful
0924 04:59:20.339108 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":98,"thread_start_us":77,"threads_started":1}
I20260924 04:59:20.339344 7206 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40695 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 04:59:20 dist-test-slave-70st krb5kdc[6442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225960, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260924 04:59:20.345525 7135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:20.339653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45708 (local address 127.0.98.61:40695)
0924 04:59:20.339690 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:20.339692 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:20.339731 (+ 39us) server_negotiation.cc:407] Connection header received
0924 04:59:20.339785 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:20.339786 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:20.339797 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:20.339839 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:20.340071 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.340538 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.341290 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.341573 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.341644 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:20.343606 (+ 1962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:20.343607 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:20.343608 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:59:20.343614 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:20.344699 (+ 1085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:20.344984 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:20.344984 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:20.344984 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:20.345024 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:20.345143 (+ 119us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:20.345143 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:20.345144 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:20.345225 (+ 81us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:20.345270 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:20.345377 (+ 107us) server_negotiation.cc:299] Negotiation successful
0924 04:59:20.345460 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 04:59:20.345548 7208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:20.339582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40695 (local address 127.0.0.1:45708)
0924 04:59:20.339694 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:20.339703 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:20.339741 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:20.339842 (+ 101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:20.339843 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:20.339942 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:20.339988 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:20.339989 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.340599 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.340599 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:20.341138 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:20.341140 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.341664 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.341665 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:20.341737 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:20.342428 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:20.342432 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:20.343529 (+ 1097us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:59:20.344768 (+ 1239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:20.344769 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:20.344769 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:20.344912 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:20.345062 (+ 150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:20.345062 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:20.345062 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:20.345105 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:20.345263 (+ 158us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:59:20.345263 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:59:20.345294 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 04:59:20.345361 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 04:59:20.345451 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98}
I20260924 04:59:20.345881 7206 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44097 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 04:59:20 dist-test-slave-70st krb5kdc[6442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225960, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260924 04:59:20.351245 7208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:20.346101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44097 (local address 127.0.0.1:47244)
0924 04:59:20.346184 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:20.346193 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:20.346236 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:20.346352 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:20.346353 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:20.346463 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:20.346486 (+ 23us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:20.346487 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.347052 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.347053 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:20.347517 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:20.347518 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.347836 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.347836 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:20.347878 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:20.348157 (+ 279us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:20.348160 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:20.349184 (+ 1024us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:59:20.350385 (+ 1201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:20.350386 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:20.350386 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:20.350532 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:20.350702 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:20.350702 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:20.350703 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:20.350752 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:20.350951 (+ 199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:59:20.350951 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:59:20.350984 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 04:59:20.351042 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 04:59:20.351125 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260924 04:59:20.351284 7209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:20.346149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47244 (local address 127.0.98.60:44097)
0924 04:59:20.346186 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:20.346188 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:20.346224 (+ 36us) server_negotiation.cc:407] Connection header received
0924 04:59:20.346292 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:20.346293 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:20.346300 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:20.346349 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:20.346542 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.346991 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.347579 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.347784 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.347833 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:20.349253 (+ 1420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:20.349254 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:20.349255 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:59:20.349260 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:20.350308 (+ 1048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:20.350584 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:20.350585 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:20.350585 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:20.350652 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:20.350809 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:20.350809 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:20.350810 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:20.350903 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:20.350951 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:20.351066 (+ 115us) server_negotiation.cc:299] Negotiation successful
0924 04:59:20.351172 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":49,"server-negotiator.queue_time_us":22}
I20260924 04:59:20.352519 7206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a65aed7f2fdc4e6ea4e8765bbe365a42: Bootstrap starting.
I20260924 04:59:20.353330 7206 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a65aed7f2fdc4e6ea4e8765bbe365a42: Neither blocks nor log segments found. Creating new log.
I20260924 04:59:20.353623 7206 log.cc:824] T 00000000000000000000000000000000 P a65aed7f2fdc4e6ea4e8765bbe365a42: Log is configured to *not* fsync() on all Append() calls
I20260924 04:59:20.354279 7206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a65aed7f2fdc4e6ea4e8765bbe365a42: No bootstrap required, opened a new log
I20260924 04:59:20.356065 7206 raft_consensus.cc:348] T 00000000000000000000000000000000 P a65aed7f2fdc4e6ea4e8765bbe365a42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "221d34b534b046078e9003aff8289b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43351 } } peers { permanent_uuid: "c82275d62aac4bc580771ab80ac00860" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40695 } } peers { permanent_uuid: "a65aed7f2fdc4e6ea4e8765bbe365a42" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44097 } }
I20260924 04:59:20.356195 7206 raft_consensus.cc:374] T 00000000000000000000000000000000 P a65aed7f2fdc4e6ea4e8765bbe365a42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:59:20.356216 7206 raft_consensus.cc:729] T 00000000000000000000000000000000 P a65aed7f2fdc4e6ea4e8765bbe365a42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a65aed7f2fdc4e6ea4e8765bbe365a42, State: Initialized, Role: FOLLOWER
I20260924 04:59:20.356325 7206 consensus_queue.cc:260] T 00000000000000000000000000000000 P a65aed7f2fdc4e6ea4e8765bbe365a42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "221d34b534b046078e9003aff8289b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43351 } } peers { permanent_uuid: "c82275d62aac4bc580771ab80ac00860" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40695 } } peers { permanent_uuid: "a65aed7f2fdc4e6ea4e8765bbe365a42" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44097 } }
I20260924 04:59:20.356608 7206 sys_catalog.cc:564] T 00000000000000000000000000000000 P a65aed7f2fdc4e6ea4e8765bbe365a42 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:59:20.356581 7211 sys_catalog.cc:455] T 00000000000000000000000000000000 P a65aed7f2fdc4e6ea4e8765bbe365a42 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "221d34b534b046078e9003aff8289b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43351 } } peers { permanent_uuid: "c82275d62aac4bc580771ab80ac00860" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40695 } } peers { permanent_uuid: "a65aed7f2fdc4e6ea4e8765bbe365a42" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44097 } } }
I20260924 04:59:20.356652 7211 sys_catalog.cc:458] T 00000000000000000000000000000000 P a65aed7f2fdc4e6ea4e8765bbe365a42 [sys.catalog]: This master's current role is: FOLLOWER
W20260924 04:59:20.358170 7222 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a65aed7f2fdc4e6ea4e8765bbe365a42: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:59:20.358227 7222 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:59:20.360234 7138 master_runner.cc:428] Detected that this master a65aed7f2fdc4e6ea4e8765bbe365a42 is joining an existing cluster
I20260924 04:59:20.360281 7138 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:44097
I20260924 04:59:20.366981 7209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:20.361380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47246 (local address 127.0.98.60:44097)
0924 04:59:20.361448 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:20.361450 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:20.361880 (+ 430us) server_negotiation.cc:407] Connection header received
0924 04:59:20.361928 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:20.361928 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:20.361937 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:20.361983 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:20.362227 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.362659 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.363666 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.363847 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.363873 (+ 26us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:20.364764 (+ 891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:20.364765 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:20.364766 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:59:20.364771 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:20.366045 (+ 1274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:20.366328 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:20.366329 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:20.366330 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:20.366377 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:20.366526 (+ 149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:20.366526 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:20.366527 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:20.366623 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:20.366673 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:20.366807 (+ 134us) server_negotiation.cc:299] Negotiation successful
0924 04:59:20.366882 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 04:59:20.367203 7229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:20.361302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44097 (local address 127.0.0.1:47246)
0924 04:59:20.361815 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:20.361824 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:20.361876 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:20.362012 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:20.362012 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:20.362129 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:20.362168 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:20.362169 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.362719 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.362720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:20.363584 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:20.363586 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.363929 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.363930 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:20.363966 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:20.364399 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:20.364403 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:20.364690 (+ 287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:59:20.366114 (+ 1424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:20.366115 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:20.366115 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:20.366270 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:20.366428 (+ 158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:20.366429 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:20.366429 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:20.366479 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:20.366670 (+ 191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:59:20.366670 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:59:20.366705 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 04:59:20.366789 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 04:59:20.366932 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"mutex_wait_us":59,"thread_start_us":58,"threads_started":1}
I20260924 04:59:20.372730 7229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:20.361688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43351 (local address 127.0.0.1:34334)
0924 04:59:20.367248 (+ 5560us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:20.367258 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:20.367301 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:20.367573 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:20.367573 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:20.367719 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:20.367757 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:20.367759 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.368648 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.368649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:20.369159 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:20.369161 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.369612 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.369612 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:20.369655 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:20.370032 (+ 377us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:20.370036 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:20.370324 (+ 288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:59:20.371835 (+ 1511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:20.371835 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:20.371835 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:20.371974 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:20.372147 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:20.372147 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:20.372148 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:20.372190 (+ 42us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:20.372434 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:59:20.372435 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:59:20.372474 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 04:59:20.372537 (+ 63us) client_negotiation.cc:240] Negotiation successful
0924 04:59:20.372606 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5552,"thread_start_us":69,"threads_started":1}
I20260924 04:59:20.372741 6527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:20.361717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34334 (local address 127.0.98.62:43351)
0924 04:59:20.361752 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:20.361754 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:20.367315 (+ 5561us) server_negotiation.cc:407] Connection header received
0924 04:59:20.367345 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:20.367346 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:20.367357 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:20.367403 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:20.367828 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.368580 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.369224 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.369553 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.369617 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:20.370405 (+ 788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:20.370406 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:20.370406 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:59:20.370414 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:20.371750 (+ 1336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:20.372038 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:20.372039 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:20.372039 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:20.372095 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:20.372229 (+ 134us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:20.372229 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:20.372230 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:20.372349 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:20.372412 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:20.372574 (+ 162us) server_negotiation.cc:299] Negotiation successful
0924 04:59:20.372645 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
Sep 24 04:59:20 dist-test-slave-70st krb5kdc[6442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225959, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260924 04:59:20.384061 7044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:20.374733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44097 (local address 127.0.0.1:47252)
0924 04:59:20.374774 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:20.374791 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:20.374823 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:20.374932 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:20.374933 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:20.375093 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:20.375132 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:20.375133 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.375658 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.375659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:20.376455 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:20.376457 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.376728 (+ 271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.376729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:20.376799 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:20.377094 (+ 295us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:20.377098 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:20.378316 (+ 1218us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:59:20.383045 (+ 4729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:20.383046 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:20.383047 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:20.383240 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:20.383444 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:20.383444 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:20.383445 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:20.383502 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:20.383751 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:59:20.383752 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:59:20.383798 (+ 46us) client_negotiation.cc:769] Sending connection context
0924 04:59:20.383867 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 04:59:20.383966 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260924 04:59:20.384104 7209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:20.374778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47252 (local address 127.0.98.60:44097)
0924 04:59:20.374809 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:20.374811 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:20.374818 (+ 7us) server_negotiation.cc:407] Connection header received
0924 04:59:20.374880 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:20.374881 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:20.374888 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:20.374928 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:20.375186 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.375599 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.376518 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.376677 (+ 159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.376717 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:20.378407 (+ 1690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:20.378409 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:20.378409 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:59:20.378415 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:20.382958 (+ 4543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:20.383314 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:20.383316 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:20.383316 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:20.383376 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:20.383563 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:20.383563 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:20.383564 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:20.383689 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:20.383754 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:20.383895 (+ 141us) server_negotiation.cc:299] Negotiation successful
0924 04:59:20.384013 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 04:59:20.384534 6487 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:44097: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "a65aed7f2fdc4e6ea4e8765bbe365a42"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 44097
}
attrs {
promote: true
}
}
dest_uuid: "221d34b534b046078e9003aff8289b32"
cas_config_opid_index: 5
I20260924 04:59:20.384788 6487 consensus_queue.cc:237] T 00000000000000000000000000000000 P 221d34b534b046078e9003aff8289b32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "221d34b534b046078e9003aff8289b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43351 } } peers { permanent_uuid: "c82275d62aac4bc580771ab80ac00860" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40695 } attrs { promote: true } } peers { permanent_uuid: "a65aed7f2fdc4e6ea4e8765bbe365a42" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44097 } attrs { promote: true } }
I20260924 04:59:20.385012 7228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:20.361093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40695 (local address 127.0.0.1:45710)
0924 04:59:20.361249 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:20.361259 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:20.361308 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:20.361574 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:20.361575 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:20.361845 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:20.361906 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:20.361908 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.362788 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.362788 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:20.363585 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:20.363586 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.364039 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.364040 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:20.364089 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:20.381855 (+ 17766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:20.381859 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:20.382203 (+ 344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:59:20.383783 (+ 1580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:20.383783 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:20.383784 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:20.384011 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:20.384377 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:20.384377 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:20.384377 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:20.384436 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:20.384714 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:59:20.384715 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:59:20.384753 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 04:59:20.384815 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 04:59:20.384920 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":85,"threads_started":1}
I20260924 04:59:20.385035 7135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:20.361404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45710 (local address 127.0.98.61:40695)
0924 04:59:20.361463 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:20.361464 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:20.361472 (+ 8us) server_negotiation.cc:407] Connection header received
0924 04:59:20.361493 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:20.361493 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:20.361504 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:20.361555 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:20.361969 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.362709 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.363661 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.363963 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.364031 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:20.382312 (+ 18281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:20.382314 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:20.382314 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:59:20.382323 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:20.383709 (+ 1386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:20.384087 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:20.384088 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:20.384088 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:20.384142 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:20.384503 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:20.384504 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:20.384504 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:20.384627 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:20.384667 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:20.384859 (+ 192us) server_negotiation.cc:299] Negotiation successful
0924 04:59:20.384944 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 04:59:20.385273 7045 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 221d34b534b046078e9003aff8289b32 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER a65aed7f2fdc4e6ea4e8765bbe365a42 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "221d34b534b046078e9003aff8289b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43351 } } peers { permanent_uuid: "c82275d62aac4bc580771ab80ac00860" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40695 } attrs { promote: true } } peers { permanent_uuid: "a65aed7f2fdc4e6ea4e8765bbe365a42" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44097 } attrs { promote: true } } }
I20260924 04:59:20.386157 7045 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:44097
I20260924 04:59:20.386216 7045 sys_catalog.cc:455] T 00000000000000000000000000000000 P 221d34b534b046078e9003aff8289b32 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "221d34b534b046078e9003aff8289b32" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "221d34b534b046078e9003aff8289b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43351 } } peers { permanent_uuid: "c82275d62aac4bc580771ab80ac00860" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40695 } attrs { promote: true } } peers { permanent_uuid: "a65aed7f2fdc4e6ea4e8765bbe365a42" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44097 } attrs { promote: true } } }
I20260924 04:59:20.386281 7045 sys_catalog.cc:458] T 00000000000000000000000000000000 P 221d34b534b046078e9003aff8289b32 [sys.catalog]: This master's current role is: LEADER
I20260924 04:59:20.386654 7180 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a65aed7f2fdc4e6ea4e8765bbe365a42 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:59:20.386739 7138 master.cc:537] Master@127.0.98.60:44097 shutting down...
I20260924 04:59:20.387558 7180 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a65aed7f2fdc4e6ea4e8765bbe365a42 [term 1 FOLLOWER]: Refusing update from remote peer 221d34b534b046078e9003aff8289b32: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260924 04:59:20.387835 7046 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 221d34b534b046078e9003aff8289b32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a65aed7f2fdc4e6ea4e8765bbe365a42" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44097 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 04:59:20.388437 6471 consensus_peers.cc:487] T 00000000000000000000000000000000 P 221d34b534b046078e9003aff8289b32 -> Peer a65aed7f2fdc4e6ea4e8765bbe365a42 (127.0.98.60:44097): Couldn't send request to peer a65aed7f2fdc4e6ea4e8765bbe365a42. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260924 04:59:20.389439 7138 raft_consensus.cc:2229] T 00000000000000000000000000000000 P a65aed7f2fdc4e6ea4e8765bbe365a42 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:59:20.389524 7138 raft_consensus.cc:2258] T 00000000000000000000000000000000 P a65aed7f2fdc4e6ea4e8765bbe365a42 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:59:20.389573 7138 tablet_replica.cc:330] T 00000000000000000000000000000000 P a65aed7f2fdc4e6ea4e8765bbe365a42: stopping tablet replica
W20260924 04:59:20.390516 6440 connection.cc:535] client connection to 127.0.98.60:44097 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 04:59:20.390909 7044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:20.386492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40695 (local address 127.0.0.1:45712)
0924 04:59:20.386556 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:20.386565 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:20.386598 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:20.386947 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:20.386948 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:20.386950 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:59:20.386988 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:20.386989 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.387904 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.387905 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:20.389944 (+ 2039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:20.389947 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.390678 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.390679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:20.390735 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:20.390754 (+ 19us) client_negotiation.cc:769] Sending connection context
0924 04:59:20.390808 (+ 54us) client_negotiation.cc:240] Negotiation successful
0924 04:59:20.390845 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260924 04:59:20.391211 7135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:20.386548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45712 (local address 127.0.98.61:40695)
0924 04:59:20.386579 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:20.386580 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:20.386587 (+ 7us) server_negotiation.cc:407] Connection header received
0924 04:59:20.386642 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:20.386642 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:20.386664 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:20.386721 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:59:20.387063 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.387739 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.390032 (+ 2293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.390619 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.390680 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:20.390756 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:20.390814 (+ 58us) server_negotiation.cc:299] Negotiation successful
0924 04:59:20.390846 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 04:59:20.391510 7094 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c82275d62aac4bc580771ab80ac00860 [term 1 LEARNER]: Refusing update from remote peer 221d34b534b046078e9003aff8289b32: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260924 04:59:20.391721 7231 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 221d34b534b046078e9003aff8289b32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c82275d62aac4bc580771ab80ac00860" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40695 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260924 04:59:20.392212 7094 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c82275d62aac4bc580771ab80ac00860 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER a65aed7f2fdc4e6ea4e8765bbe365a42 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "221d34b534b046078e9003aff8289b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43351 } } peers { permanent_uuid: "c82275d62aac4bc580771ab80ac00860" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40695 } attrs { promote: true } } peers { permanent_uuid: "a65aed7f2fdc4e6ea4e8765bbe365a42" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44097 } attrs { promote: true } } }
I20260924 04:59:20.393036 7233 sys_catalog.cc:455] T 00000000000000000000000000000000 P c82275d62aac4bc580771ab80ac00860 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "221d34b534b046078e9003aff8289b32" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "221d34b534b046078e9003aff8289b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43351 } } peers { permanent_uuid: "c82275d62aac4bc580771ab80ac00860" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40695 } attrs { promote: true } } peers { permanent_uuid: "a65aed7f2fdc4e6ea4e8765bbe365a42" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44097 } attrs { promote: true } } }
I20260924 04:59:20.393131 7233 sys_catalog.cc:458] T 00000000000000000000000000000000 P c82275d62aac4bc580771ab80ac00860 [sys.catalog]: This master's current role is: LEARNER
I20260924 04:59:20.393270 7123 sys_catalog.cc:455] T 00000000000000000000000000000000 P c82275d62aac4bc580771ab80ac00860 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 221d34b534b046078e9003aff8289b32. Latest consensus state: current_term: 1 leader_uuid: "221d34b534b046078e9003aff8289b32" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "221d34b534b046078e9003aff8289b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43351 } } peers { permanent_uuid: "c82275d62aac4bc580771ab80ac00860" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40695 } attrs { promote: true } } peers { permanent_uuid: "a65aed7f2fdc4e6ea4e8765bbe365a42" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44097 } attrs { promote: true } } }
I20260924 04:59:20.393343 7123 sys_catalog.cc:458] T 00000000000000000000000000000000 P c82275d62aac4bc580771ab80ac00860 [sys.catalog]: This master's current role is: LEARNER
I20260924 04:59:20.393533 7046 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 221d34b534b046078e9003aff8289b32: attempting to promote NON_VOTER c82275d62aac4bc580771ab80ac00860 to VOTER
I20260924 04:59:20.393721 7046 consensus_queue.cc:237] T 00000000000000000000000000000000 P 221d34b534b046078e9003aff8289b32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "221d34b534b046078e9003aff8289b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43351 } } peers { permanent_uuid: "c82275d62aac4bc580771ab80ac00860" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40695 } attrs { promote: false } } peers { permanent_uuid: "a65aed7f2fdc4e6ea4e8765bbe365a42" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44097 } attrs { promote: true } }
I20260924 04:59:20.394150 7044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:20.394045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44097 (local address 127.0.0.1:47262)
0924 04:59:20.394083 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:20.394125 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260924 04:59:20.394198 7094 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c82275d62aac4bc580771ab80ac00860 [term 1 LEARNER]: Refusing update from remote peer 221d34b534b046078e9003aff8289b32: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260924 04:59:20.394507 7045 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 221d34b534b046078e9003aff8289b32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c82275d62aac4bc580771ab80ac00860" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40695 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260924 04:59:20.394567 7234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:20.394342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44097 (local address 127.0.0.1:47264)
0924 04:59:20.394478 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:20.394509 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":85,"threads_started":1}
W20260924 04:59:20.394680 6471 consensus_peers.cc:487] T 00000000000000000000000000000000 P 221d34b534b046078e9003aff8289b32 -> Peer a65aed7f2fdc4e6ea4e8765bbe365a42 (127.0.98.60:44097): Couldn't send request to peer a65aed7f2fdc4e6ea4e8765bbe365a42. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44097: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 04:59:20.395169 7231 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 221d34b534b046078e9003aff8289b32 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c82275d62aac4bc580771ab80ac00860 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "221d34b534b046078e9003aff8289b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43351 } } peers { permanent_uuid: "c82275d62aac4bc580771ab80ac00860" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40695 } attrs { promote: false } } peers { permanent_uuid: "a65aed7f2fdc4e6ea4e8765bbe365a42" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44097 } attrs { promote: true } } }
I20260924 04:59:20.395375 7094 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c82275d62aac4bc580771ab80ac00860 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c82275d62aac4bc580771ab80ac00860 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "221d34b534b046078e9003aff8289b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43351 } } peers { permanent_uuid: "c82275d62aac4bc580771ab80ac00860" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40695 } attrs { promote: false } } peers { permanent_uuid: "a65aed7f2fdc4e6ea4e8765bbe365a42" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44097 } attrs { promote: true } } }
I20260924 04:59:20.395893 7045 sys_catalog.cc:455] T 00000000000000000000000000000000 P 221d34b534b046078e9003aff8289b32 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "221d34b534b046078e9003aff8289b32" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "221d34b534b046078e9003aff8289b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43351 } } peers { permanent_uuid: "c82275d62aac4bc580771ab80ac00860" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40695 } attrs { promote: false } } peers { permanent_uuid: "a65aed7f2fdc4e6ea4e8765bbe365a42" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44097 } attrs { promote: true } } }
I20260924 04:59:20.395982 7045 sys_catalog.cc:458] T 00000000000000000000000000000000 P 221d34b534b046078e9003aff8289b32 [sys.catalog]: This master's current role is: LEADER
I20260924 04:59:20.395895 7231 sys_catalog.cc:455] T 00000000000000000000000000000000 P 221d34b534b046078e9003aff8289b32 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "221d34b534b046078e9003aff8289b32" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "221d34b534b046078e9003aff8289b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43351 } } peers { permanent_uuid: "c82275d62aac4bc580771ab80ac00860" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40695 } attrs { promote: false } } peers { permanent_uuid: "a65aed7f2fdc4e6ea4e8765bbe365a42" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44097 } attrs { promote: true } } }
I20260924 04:59:20.396097 7231 sys_catalog.cc:458] T 00000000000000000000000000000000 P 221d34b534b046078e9003aff8289b32 [sys.catalog]: This master's current role is: LEADER
I20260924 04:59:20.396162 7123 sys_catalog.cc:455] T 00000000000000000000000000000000 P c82275d62aac4bc580771ab80ac00860 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "221d34b534b046078e9003aff8289b32" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "221d34b534b046078e9003aff8289b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43351 } } peers { permanent_uuid: "c82275d62aac4bc580771ab80ac00860" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40695 } attrs { promote: false } } peers { permanent_uuid: "a65aed7f2fdc4e6ea4e8765bbe365a42" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44097 } attrs { promote: true } } }
I20260924 04:59:20.396227 7123 sys_catalog.cc:458] T 00000000000000000000000000000000 P c82275d62aac4bc580771ab80ac00860 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:59:20.402372 7138 master.cc:552] Master@127.0.98.60:44097 shutdown complete.
I20260924 04:59:20.402423 7138 master_runner.cc:305] Clearing existing system tablet
I20260924 04:59:20.402540 7138 fs_manager.cc:359] Metadata directory not provided
I20260924 04:59:20.402582 7138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:59:20.403216 7138 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260924 04:59:20.403664 7236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:20.403774 7138 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:59:20.403836 7138 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:20.403858 7138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/master-2/wal
uuid: "a65aed7f2fdc4e6ea4e8765bbe365a42"
format_stamp: "Formatted at 2026-09-24 04:59:20 on dist-test-slave-70st"
server_key: "5173dee5f88860e4d6b8973925341859"
server_key_iv: "189e7ab7c894a27ae90eb5bf4d07c108"
server_key_version: "encryptionkey@0"
I20260924 04:59:20.404588 7138 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P a65aed7f2fdc4e6ea4e8765bbe365a42: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:59:20.405910 7138 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P a65aed7f2fdc4e6ea4e8765bbe365a42: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 04:59:20.405974 7138 log.cc:1196] T 00000000000000000000000000000000 P a65aed7f2fdc4e6ea4e8765bbe365a42: Deleting WAL directory at /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260924 04:59:20.406150 7138 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P a65aed7f2fdc4e6ea4e8765bbe365a42: Deleting consensus metadata
I20260924 04:59:20.406433 7138 master_runner.cc:315] Copying system tablet from 127.0.98.62:43351
I20260924 04:59:20.407013 7138 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P a65aed7f2fdc4e6ea4e8765bbe365a42: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:43351
I20260924 04:59:20.412292 6527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:20.407335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34348 (local address 127.0.98.62:43351)
0924 04:59:20.407378 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:20.407380 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:20.407482 (+ 102us) server_negotiation.cc:407] Connection header received
0924 04:59:20.407540 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:20.407541 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:20.407550 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:20.407594 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:20.407886 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.408359 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.409029 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.409226 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.409278 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:20.410227 (+ 949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:20.410229 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:20.410229 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:59:20.410235 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:20.411351 (+ 1116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:20.411637 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:20.411638 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:20.411638 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:20.411690 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:20.411840 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:20.411841 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:20.411841 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:20.411954 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:20.412009 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:20.412137 (+ 128us) server_negotiation.cc:299] Negotiation successful
0924 04:59:20.412204 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 04:59:20.412310 7242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:20.407254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43351 (local address 127.0.0.1:34348)
0924 04:59:20.407420 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:20.407429 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:20.407485 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:20.407597 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:20.407598 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:20.407765 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:20.407815 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:20.407816 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.408426 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.408426 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:20.408959 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:20.408961 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.409281 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.409282 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:20.409338 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:20.409884 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:20.409888 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:20.410152 (+ 264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:59:20.411422 (+ 1270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:20.411423 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:20.411424 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:20.411579 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:20.411740 (+ 161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:20.411741 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:20.411741 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:20.411792 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:20.412008 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:59:20.412009 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:59:20.412052 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 04:59:20.412111 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 04:59:20.412204 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":66,"threads_started":1}
I20260924 04:59:20.412647 6507 tablet_copy_service.cc:140] P 221d34b534b046078e9003aff8289b32: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer a65aed7f2fdc4e6ea4e8765bbe365a42 ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:34348)
I20260924 04:59:20.412698 6507 tablet_copy_service.cc:161] P 221d34b534b046078e9003aff8289b32: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer a65aed7f2fdc4e6ea4e8765bbe365a42 at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:34348: session id = a65aed7f2fdc4e6ea4e8765bbe365a42-00000000000000000000000000000000
I20260924 04:59:20.413096 6507 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 221d34b534b046078e9003aff8289b32: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 04:59:20.413540 7138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 04:59:20.414690 7138 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P a65aed7f2fdc4e6ea4e8765bbe365a42: tablet copy: Starting download of 0 data blocks...
I20260924 04:59:20.414845 7138 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P a65aed7f2fdc4e6ea4e8765bbe365a42: tablet copy: Starting download of 1 WAL segments...
I20260924 04:59:20.415545 7138 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P a65aed7f2fdc4e6ea4e8765bbe365a42: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 04:59:20.416345 6507 tablet_copy_service.cc:342] P 221d34b534b046078e9003aff8289b32: Request end of tablet copy session a65aed7f2fdc4e6ea4e8765bbe365a42-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:34348
I20260924 04:59:20.416399 6507 tablet_copy_service.cc:433] P 221d34b534b046078e9003aff8289b32: ending tablet copy session a65aed7f2fdc4e6ea4e8765bbe365a42-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer a65aed7f2fdc4e6ea4e8765bbe365a42
I20260924 04:59:20.418672 7138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:59:20.419742 7244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:59:20.419747 7246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:59:20.419749 7243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:59:20.421567 7138 server_base.cc:1034] running on GCE node
I20260924 04:59:20.421644 7138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:59:20.421767 7138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:59:20.422878 7138 hybrid_clock.cc:648] HybridClock initialized: now 1790225960422873 us; error 24 us; skew 500 ppm
Sep 24 04:59:20 dist-test-slave-70st krb5kdc[6442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225960, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:59:20.424186 7138 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260924 04:59:20.424413 7138 webserver.cc:460] Webserver started at http://127.0.98.60:44351/ using document root <none> and password file <none>
I20260924 04:59:20.424522 7138 fs_manager.cc:359] Metadata directory not provided
I20260924 04:59:20.424553 7138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:59:20.425122 7138 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260924 04:59:20.425591 7253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:20.425715 7138 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:20.425769 7138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/master-2/wal
uuid: "a65aed7f2fdc4e6ea4e8765bbe365a42"
format_stamp: "Formatted at 2026-09-24 04:59:20 on dist-test-slave-70st"
server_key: "5173dee5f88860e4d6b8973925341859"
server_key_iv: "189e7ab7c894a27ae90eb5bf4d07c108"
server_key_version: "encryptionkey@0"
I20260924 04:59:20.425832 7138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:59:20.443192 7138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:59:20.444680 7138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:59:20.448182 7138 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:44097
I20260924 04:59:20.448354 7305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:44097 every 8 connection(s)
I20260924 04:59:20.448541 7138 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/master-2/data/info.pb
I20260924 04:59:20.449235 7306 sys_catalog.cc:263] Verifying existing consensus state
I20260924 04:59:20.449646 7306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a65aed7f2fdc4e6ea4e8765bbe365a42: Bootstrap starting.
I20260924 04:59:20.451826 7306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a65aed7f2fdc4e6ea4e8765bbe365a42: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 04:59:20.451990 7306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a65aed7f2fdc4e6ea4e8765bbe365a42: Bootstrap complete.
I20260924 04:59:20.452159 7306 raft_consensus.cc:348] T 00000000000000000000000000000000 P a65aed7f2fdc4e6ea4e8765bbe365a42 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "221d34b534b046078e9003aff8289b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43351 } } peers { permanent_uuid: "c82275d62aac4bc580771ab80ac00860" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40695 } attrs { promote: false } } peers { permanent_uuid: "a65aed7f2fdc4e6ea4e8765bbe365a42" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44097 } attrs { promote: true } }
I20260924 04:59:20.452235 7306 raft_consensus.cc:729] T 00000000000000000000000000000000 P a65aed7f2fdc4e6ea4e8765bbe365a42 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a65aed7f2fdc4e6ea4e8765bbe365a42, State: Initialized, Role: LEARNER
I20260924 04:59:20.452261 7306 consensus_queue.cc:260] T 00000000000000000000000000000000 P a65aed7f2fdc4e6ea4e8765bbe365a42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "221d34b534b046078e9003aff8289b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43351 } } peers { permanent_uuid: "c82275d62aac4bc580771ab80ac00860" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40695 } attrs { promote: false } } peers { permanent_uuid: "a65aed7f2fdc4e6ea4e8765bbe365a42" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44097 } attrs { promote: true } }
I20260924 04:59:20.452428 7306 sys_catalog.cc:564] T 00000000000000000000000000000000 P a65aed7f2fdc4e6ea4e8765bbe365a42 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:59:20.452420 7309 sys_catalog.cc:455] T 00000000000000000000000000000000 P a65aed7f2fdc4e6ea4e8765bbe365a42 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "221d34b534b046078e9003aff8289b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43351 } } peers { permanent_uuid: "c82275d62aac4bc580771ab80ac00860" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40695 } attrs { promote: false } } peers { permanent_uuid: "a65aed7f2fdc4e6ea4e8765bbe365a42" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44097 } attrs { promote: true } } }
I20260924 04:59:20.452522 7309 sys_catalog.cc:458] T 00000000000000000000000000000000 P a65aed7f2fdc4e6ea4e8765bbe365a42 [sys.catalog]: This master's current role is: LEARNER
I20260924 04:59:20.454072 7320 catalog_manager.cc:1259] Loaded cluster ID: c56b85dc80bd450c8c3816861b8b9cb0
I20260924 04:59:20.454113 7320 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a65aed7f2fdc4e6ea4e8765bbe365a42: loading cluster ID for follower catalog manager: success
I20260924 04:59:20.455009 7320 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a65aed7f2fdc4e6ea4e8765bbe365a42: acquiring CA information for follower catalog manager: success
I20260924 04:59:20.455257 7320 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a65aed7f2fdc4e6ea4e8765bbe365a42: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 04:59:20.497058 7321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:20.491493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47286 (local address 127.0.98.60:44097)
0924 04:59:20.491617 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:20.491619 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:20.491625 (+ 6us) server_negotiation.cc:407] Connection header received
0924 04:59:20.491647 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:20.491647 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:20.491656 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:20.491707 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:20.492211 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.492822 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.493521 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.493706 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.493756 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:20.494830 (+ 1074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:20.494831 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:20.494831 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:59:20.494838 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:20.496086 (+ 1248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:20.496375 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:20.496376 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:20.496376 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:20.496428 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:20.496588 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:20.496588 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:20.496589 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:20.496686 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:20.496767 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:20.496892 (+ 125us) server_negotiation.cc:299] Negotiation successful
0924 04:59:20.496963 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":61,"threads_started":1}
I20260924 04:59:20.497521 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 6549
I20260924 04:59:20.501451 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:46631
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:43351
--webserver_port=43981
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:43351,127.0.98.61:40695
--tserver_master_addrs=127.0.98.62:43351,127.0.98.61:40695,127.0.98.60:44097 with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:59:20.579218 7322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:59:20.579429 7322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:59:20.579461 7322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:59:20.580554 7322 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:59:20.580591 7322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:59:20.580626 7322 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:59:20.582041 7322 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:46631
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=43981
--tserver_master_addrs=127.0.98.62:43351,127.0.98.61:40695,127.0.98.60:44097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7322
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 04:59:20.582267 7322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:59:20.582471 7322 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:59:20.587077 7328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:59:20.587111 7322 server_base.cc:1034] running on GCE node
W20260924 04:59:20.587124 7327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:59:20.587080 7330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:59:20.587469 7322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:59:20.587680 7322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:59:20.588814 7322 hybrid_clock.cc:648] HybridClock initialized: now 1790225960588805 us; error 33 us; skew 500 ppm
Sep 24 04:59:20 dist-test-slave-70st krb5kdc[6442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225960, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:59:20.591146 7322 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260924 04:59:20.591550 7322 webserver.cc:460] Webserver started at http://127.0.98.1:43981/ using document root <none> and password file <none>
I20260924 04:59:20.591768 7322 fs_manager.cc:359] Metadata directory not provided
I20260924 04:59:20.591831 7322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:59:20.593051 7322 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 04:59:20.593576 7337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:20.593705 7322 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:20.593780 7322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/ts-0/wal
uuid: "2b2dd144b90544c5bdbefde04c6672d5"
format_stamp: "Formatted at 2026-09-24 04:59:19 on dist-test-slave-70st"
server_key: "77f12540beb6d735f20a79451546b095"
server_key_iv: "c15f5e8e61f6ff60cb7d4aa49f8aaabc"
server_key_version: "encryptionkey@0"
I20260924 04:59:20.594089 7322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:59:20.608788 7322 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:59:20.610852 7322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:59:20.610983 7322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:59:20.611171 7322 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:59:20.611435 7322 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:59:20.611465 7322 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:20.611493 7322 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:59:20.611516 7322 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:20.617029 7322 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46631
I20260924 04:59:20.617080 7450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46631 every 8 connection(s)
I20260924 04:59:20.617391 7322 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 04:59:20 dist-test-slave-70st krb5kdc[6442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225960, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 24 04:59:20 dist-test-slave-70st krb5kdc[6442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225960, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 24 04:59:20 dist-test-slave-70st krb5kdc[6442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225960, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 04:59:20.625602 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 7322
I20260924 04:59:20.625768 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 6683
I20260924 04:59:20.627414 7135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:20.618429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40255 (local address 127.0.98.61:40695)
0924 04:59:20.618474 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:20.618476 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:20.619360 (+ 884us) server_negotiation.cc:407] Connection header received
0924 04:59:20.619391 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:20.619393 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:20.619403 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:20.619456 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:20.619776 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.620260 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.621641 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.621862 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.621917 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:20.624561 (+ 2644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:20.624562 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:20.624562 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:59:20.624569 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:20.626193 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:20.626595 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:20.626596 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:20.626597 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:20.626649 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:20.626821 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:20.626821 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:20.626822 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:20.626947 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:20.626984 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:20.627226 (+ 242us) server_negotiation.cc:299] Negotiation successful
0924 04:59:20.627298 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 04:59:20.627413 7321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:20.618082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56545 (local address 127.0.98.60:44097)
0924 04:59:20.618162 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:20.618164 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:20.618940 (+ 776us) server_negotiation.cc:407] Connection header received
0924 04:59:20.619024 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:20.619025 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:20.619037 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:20.619090 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:20.619753 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.620519 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.621701 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.622012 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.622077 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:20.624145 (+ 2068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:20.624146 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:20.624147 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:59:20.624152 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:20.625881 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:20.626389 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:20.626390 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:20.626390 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:20.626446 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:20.626709 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:20.626709 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:20.626710 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:20.626835 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:20.626876 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:20.627232 (+ 356us) server_negotiation.cc:299] Negotiation successful
0924 04:59:20.627324 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 04:59:20.627707 6527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:20.618105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43227 (local address 127.0.98.62:43351)
0924 04:59:20.618157 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:20.618160 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:20.618346 (+ 186us) server_negotiation.cc:407] Connection header received
0924 04:59:20.618870 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:20.618872 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:20.618882 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:20.618940 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:20.619784 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.620519 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.622354 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.622593 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.622654 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:20.625023 (+ 2369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:20.625024 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:20.625025 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:59:20.625032 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:20.626612 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:20.626941 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:20.626942 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:20.626942 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:20.627010 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:20.627205 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:20.627206 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:20.627206 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:20.627319 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:20.627357 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:20.627558 (+ 201us) server_negotiation.cc:299] Negotiation successful
0924 04:59:20.627624 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 04:59:20.627900 7456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:20.618062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44097 (local address 127.0.98.1:56545)
0924 04:59:20.618896 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:20.618903 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:20.618966 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:20.619133 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:20.619134 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:20.619416 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:20.619677 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:20.619682 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.620595 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.620596 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:20.621593 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:20.621596 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.622082 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.622083 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:20.622144 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:20.622605 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:20.622615 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:20.624067 (+ 1452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:59:20.625965 (+ 1898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:20.625972 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:20.625974 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:20.626310 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:20.626513 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:20.626513 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:20.626513 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:20.626640 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:20.626921 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:59:20.626927 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:59:20.627088 (+ 161us) client_negotiation.cc:769] Sending connection context
0924 04:59:20.627528 (+ 440us) client_negotiation.cc:240] Negotiation successful
0924 04:59:20.627689 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":810,"mutex_wait_us":51,"thread_start_us":87,"threads_started":1}
I20260924 04:59:20.628130 7455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:20.618061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43351 (local address 127.0.98.1:43227)
0924 04:59:20.618223 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:20.618261 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:20.618757 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:20.619035 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:20.619039 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:20.619445 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:20.619677 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:20.619682 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.621561 (+ 1879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.621561 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:20.622282 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:20.622285 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.622666 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.622667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:20.622730 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:20.623126 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:20.623130 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:20.624946 (+ 1816us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:59:20.626682 (+ 1736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:20.626682 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:20.626683 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:20.626868 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:20.627088 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:20.627089 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:20.627089 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:20.627139 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:20.627400 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:59:20.627401 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:59:20.627449 (+ 48us) client_negotiation.cc:769] Sending connection context
0924 04:59:20.627797 (+ 348us) client_negotiation.cc:240] Negotiation successful
0924 04:59:20.627872 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":24,"thread_start_us":78,"threads_started":1}
I20260924 04:59:20.628607 7457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:20.618348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40695 (local address 127.0.98.1:40255)
0924 04:59:20.619290 (+ 942us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:20.619299 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:20.619351 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:20.619471 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:20.619472 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:20.619601 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:20.619677 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:20.619682 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.620351 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.620352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:20.621558 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:20.621560 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.621929 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.621930 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:20.622067 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:20.622656 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:20.622661 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:20.624492 (+ 1831us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:59:20.626317 (+ 1825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:20.626318 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:20.626319 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:20.626520 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:20.626712 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:20.626712 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:20.626713 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:20.626762 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:20.627024 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:59:20.627025 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:59:20.627096 (+ 71us) client_negotiation.cc:769] Sending connection context
0924 04:59:20.628400 (+ 1304us) client_negotiation.cc:240] Negotiation successful
0924 04:59:20.628498 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":685,"mutex_wait_us":1,"thread_start_us":54,"threads_started":1}
I20260924 04:59:20.631301 7453 heartbeater.cc:347] Connected to a master server at 127.0.98.61:40695
I20260924 04:59:20.631408 7453 heartbeater.cc:464] Registering TS with master...
I20260924 04:59:20.631585 7453 heartbeater.cc:511] Master 127.0.98.61:40695 requested a full tablet report, sending...
I20260924 04:59:20.632081 7084 ts_manager.cc:194] Registered new tserver with Master: 2b2dd144b90544c5bdbefde04c6672d5 (127.0.98.1:46631)
I20260924 04:59:20.632166 7452 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43351
I20260924 04:59:20.632221 7452 heartbeater.cc:464] Registering TS with master...
I20260924 04:59:20.632284 7452 heartbeater.cc:511] Master 127.0.98.62:43351 requested a full tablet report, sending...
I20260924 04:59:20.632570 6487 ts_manager.cc:194] Re-registered known tserver with Master: 2b2dd144b90544c5bdbefde04c6672d5 (127.0.98.1:46631)
I20260924 04:59:20.633019 6487 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:43227
I20260924 04:59:20.633122 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:42241
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:43351
--webserver_port=38481
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:43351,127.0.98.61:40695
--tserver_master_addrs=127.0.98.62:43351,127.0.98.61:40695,127.0.98.60:44097 with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:59:20.633481 7451 heartbeater.cc:347] Connected to a master server at 127.0.98.60:44097
I20260924 04:59:20.633529 7451 heartbeater.cc:464] Registering TS with master...
I20260924 04:59:20.633708 7451 heartbeater.cc:511] Master 127.0.98.60:44097 requested a full tablet report, sending...
I20260924 04:59:20.634025 7270 ts_manager.cc:194] Registered new tserver with Master: 2b2dd144b90544c5bdbefde04c6672d5 (127.0.98.1:46631)
W20260924 04:59:20.711395 7458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:59:20.711596 7458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:59:20.711625 7458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:59:20.712718 7458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:59:20.712756 7458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:59:20.712790 7458 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:59:20.714280 7458 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:42241
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=38481
--tserver_master_addrs=127.0.98.62:43351,127.0.98.61:40695,127.0.98.60:44097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7458
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 04:59:20.714511 7458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:59:20.714733 7458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:59:20.719576 7458 server_base.cc:1034] running on GCE node
W20260924 04:59:20.719544 7463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:59:20.719560 7464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:59:20.719560 7466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:59:20.719931 7458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:59:20.720170 7458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:59:20.721302 7458 hybrid_clock.cc:648] HybridClock initialized: now 1790225960721289 us; error 29 us; skew 500 ppm
Sep 24 04:59:20 dist-test-slave-70st krb5kdc[6442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225960, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:59:20.723652 7458 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260924 04:59:20.724097 7458 webserver.cc:460] Webserver started at http://127.0.98.2:38481/ using document root <none> and password file <none>
I20260924 04:59:20.724289 7458 fs_manager.cc:359] Metadata directory not provided
I20260924 04:59:20.724349 7458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:59:20.725725 7458 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 04:59:20.726298 7473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:20.726490 7458 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:20.726572 7458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/ts-1/wal
uuid: "fc45fc89660947038800b5a9922ea04c"
format_stamp: "Formatted at 2026-09-24 04:59:19 on dist-test-slave-70st"
server_key: "4d10af10afb7282764a00aa1b31791cd"
server_key_iv: "c7d57dbe70cb6e64f34416d19ef4fefd"
server_key_version: "encryptionkey@0"
I20260924 04:59:20.726913 7458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:59:20.747720 7458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:59:20.749614 7458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:59:20.749745 7458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:59:20.749960 7458 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:59:20.750237 7458 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:59:20.750268 7458 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:20.750296 7458 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:59:20.750320 7458 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:20.755659 7458 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42241
I20260924 04:59:20.755712 7586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42241 every 8 connection(s)
I20260924 04:59:20.756039 7458 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:59:20.757673 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 7458
I20260924 04:59:20.757797 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 6817
Sep 24 04:59:20 dist-test-slave-70st krb5kdc[6442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225960, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260924 04:59:20.764828 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:44049
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:43351
--webserver_port=37593
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:43351,127.0.98.61:40695
--tserver_master_addrs=127.0.98.62:43351,127.0.98.61:40695,127.0.98.60:44097 with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 04:59:20 dist-test-slave-70st krb5kdc[6442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225960, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 24 04:59:20 dist-test-slave-70st krb5kdc[6442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225960, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 04:59:20.767822 7321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:20.756709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42301 (local address 127.0.98.60:44097)
0924 04:59:20.756801 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:20.756803 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:20.756984 (+ 181us) server_negotiation.cc:407] Connection header received
0924 04:59:20.758558 (+ 1574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:20.758560 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:20.758572 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:20.758602 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:20.759637 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.760397 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.762013 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.762304 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.762340 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:20.764981 (+ 2641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:20.764983 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:20.764983 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:59:20.764990 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:20.766434 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:20.766862 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:20.766863 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:20.766863 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:20.766916 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:20.767128 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:20.767128 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:20.767129 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:20.767236 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:20.767291 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:20.767649 (+ 358us) server_negotiation.cc:299] Negotiation successful
0924 04:59:20.767729 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 04:59:20.768368 7591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:20.756708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44097 (local address 127.0.98.2:42301)
0924 04:59:20.756863 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:20.756898 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:20.758440 (+ 1542us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:20.758778 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:20.758782 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:20.759240 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:20.759537 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:20.759542 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.760525 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.760525 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:20.761920 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:20.761922 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.762418 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.762418 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:20.762507 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:20.763182 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:20.763185 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:20.764874 (+ 1689us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:59:20.766505 (+ 1631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:20.766511 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:20.766513 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:20.766777 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:20.766970 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:20.766971 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:20.766971 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:20.767074 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:20.767315 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:59:20.767322 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:59:20.767492 (+ 170us) client_negotiation.cc:769] Sending connection context
0924 04:59:20.767952 (+ 460us) client_negotiation.cc:240] Negotiation successful
0924 04:59:20.768139 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"mutex_wait_us":126,"thread_start_us":87,"threads_started":1}
I20260924 04:59:20.768993 7587 heartbeater.cc:347] Connected to a master server at 127.0.98.60:44097
I20260924 04:59:20.769099 7587 heartbeater.cc:464] Registering TS with master...
I20260924 04:59:20.769140 6527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:20.757309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40413 (local address 127.0.98.62:43351)
0924 04:59:20.757363 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:20.757365 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:20.757373 (+ 8us) server_negotiation.cc:407] Connection header received
0924 04:59:20.758570 (+ 1197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:20.758571 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:20.758584 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:20.758613 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:20.759638 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.760381 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.762006 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.762254 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.762293 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:20.765935 (+ 3642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:20.765937 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:20.765937 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:59:20.765945 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:20.767870 (+ 1925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:20.768351 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:20.768352 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:20.768352 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:20.768410 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:20.768595 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:20.768596 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:20.768596 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:20.768734 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:20.768790 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:20.768977 (+ 187us) server_negotiation.cc:299] Negotiation successful
0924 04:59:20.769050 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 04:59:20.769256 7135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:20.757040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43693 (local address 127.0.98.61:40695)
0924 04:59:20.757080 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:20.757082 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:20.757095 (+ 13us) server_negotiation.cc:407] Connection header received
0924 04:59:20.758560 (+ 1465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:20.758561 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:20.758572 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:20.758602 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:20.759639 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.760392 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.762013 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.762304 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.762340 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:20.766608 (+ 4268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:20.766610 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:20.766610 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:59:20.766617 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:20.768070 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:20.768460 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:20.768462 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:20.768462 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:20.768535 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:20.768736 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:20.768737 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:20.768737 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:20.768860 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:20.768922 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:20.769103 (+ 181us) server_negotiation.cc:299] Negotiation successful
0924 04:59:20.769177 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 04:59:20.769289 7587 heartbeater.cc:511] Master 127.0.98.60:44097 requested a full tablet report, sending...
I20260924 04:59:20.769559 7593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:20.757138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43351 (local address 127.0.98.2:40413)
0924 04:59:20.757260 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:20.757270 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:20.758447 (+ 1177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:20.758784 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:20.758785 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:20.759258 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:20.759538 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:20.759543 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.760532 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.760533 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:20.761919 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:20.761921 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.762375 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.762375 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:20.762506 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:20.763296 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:20.763301 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:20.765840 (+ 2539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:59:20.767968 (+ 2128us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:20.767969 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:20.767969 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:20.768271 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:20.768470 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:20.768470 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:20.768471 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:20.768528 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:20.768831 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:59:20.768831 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:59:20.768872 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 04:59:20.769263 (+ 391us) client_negotiation.cc:240] Negotiation successful
0924 04:59:20.769437 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"mutex_wait_us":113,"thread_start_us":58,"threads_started":1}
I20260924 04:59:20.769629 7592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:20.756962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40695 (local address 127.0.98.2:43693)
0924 04:59:20.757055 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:20.757063 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:20.758448 (+ 1385us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:20.758778 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:20.758782 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:20.759251 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:20.759537 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:20.759542 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.760525 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.760525 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:20.761920 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:20.761922 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.762413 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.762413 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:20.762507 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:20.763106 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:20.763114 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:20.766520 (+ 3406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:59:20.768150 (+ 1630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:20.768151 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:20.768151 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:20.768377 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:20.768610 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:20.768611 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:20.768611 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:20.768667 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:20.768956 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:59:20.768957 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:59:20.768998 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 04:59:20.769330 (+ 332us) client_negotiation.cc:240] Negotiation successful
0924 04:59:20.769492 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78,"mutex_wait_us":186,"thread_start_us":56,"threads_started":1}
I20260924 04:59:20.769734 7270 ts_manager.cc:194] Registered new tserver with Master: fc45fc89660947038800b5a9922ea04c (127.0.98.2:42241)
I20260924 04:59:20.769912 7588 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43351
I20260924 04:59:20.769959 7588 heartbeater.cc:464] Registering TS with master...
I20260924 04:59:20.770025 7588 heartbeater.cc:511] Master 127.0.98.62:43351 requested a full tablet report, sending...
I20260924 04:59:20.770170 7589 heartbeater.cc:347] Connected to a master server at 127.0.98.61:40695
I20260924 04:59:20.770207 7589 heartbeater.cc:464] Registering TS with master...
I20260924 04:59:20.770221 6487 ts_manager.cc:194] Re-registered known tserver with Master: fc45fc89660947038800b5a9922ea04c (127.0.98.2:42241)
I20260924 04:59:20.770255 7589 heartbeater.cc:511] Master 127.0.98.61:40695 requested a full tablet report, sending...
I20260924 04:59:20.770495 7084 ts_manager.cc:194] Registered new tserver with Master: fc45fc89660947038800b5a9922ea04c (127.0.98.2:42241)
I20260924 04:59:20.770684 6487 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:40413
W20260924 04:59:20.848428 7594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:59:20.848639 7594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:59:20.848671 7594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:59:20.849812 7594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:59:20.849849 7594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:59:20.849884 7594 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:59:20.851310 7594 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:44049
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=37593
--tserver_master_addrs=127.0.98.62:43351,127.0.98.61:40695,127.0.98.60:44097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7594
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 04:59:20.851537 7594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:59:20.851761 7594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:59:20.856664 7594 server_base.cc:1034] running on GCE node
W20260924 04:59:20.856649 7602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:59:20.856640 7599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:59:20.856649 7600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:59:20.856992 7594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:59:20.857223 7594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:59:20.858353 7594 hybrid_clock.cc:648] HybridClock initialized: now 1790225960858329 us; error 39 us; skew 500 ppm
Sep 24 04:59:20 dist-test-slave-70st krb5kdc[6442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225960, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:59:20.860935 7594 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260924 04:59:20.861465 7594 webserver.cc:460] Webserver started at http://127.0.98.3:37593/ using document root <none> and password file <none>
I20260924 04:59:20.861701 7594 fs_manager.cc:359] Metadata directory not provided
I20260924 04:59:20.861771 7594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:59:20.863500 7594 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 04:59:20.864161 7609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:20.864313 7594 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 04:59:20.864391 7594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/ts-2/wal
uuid: "b08a35b7530a47f48a8a4cf653551a6a"
format_stamp: "Formatted at 2026-09-24 04:59:19 on dist-test-slave-70st"
server_key: "971d9f66deaadd968a5419da172d60e6"
server_key_iv: "1a109846bbbe47e03c431940d2cfbe08"
server_key_version: "encryptionkey@0"
I20260924 04:59:20.864712 7594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:59:20.874447 7594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:59:20.876499 7594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:59:20.876627 7594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:59:20.876816 7594 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:59:20.877103 7594 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:59:20.877136 7594 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:20.877164 7594 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:59:20.877188 7594 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:20.882369 7594 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44049
I20260924 04:59:20.882433 7722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44049 every 8 connection(s)
I20260924 04:59:20.882717 7594 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 04:59:20 dist-test-slave-70st krb5kdc[6442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225960, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 24 04:59:20 dist-test-slave-70st krb5kdc[6442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225960, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 24 04:59:20 dist-test-slave-70st krb5kdc[6442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225960, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260924 04:59:20.890661 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 7594
I20260924 04:59:20.894469 7135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:20.883574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55805 (local address 127.0.98.61:40695)
0924 04:59:20.883617 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:20.883619 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:20.883731 (+ 112us) server_negotiation.cc:407] Connection header received
0924 04:59:20.884692 (+ 961us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:20.884694 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:20.884704 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:20.884762 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:20.886948 (+ 2186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.887582 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.888344 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.888627 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.888692 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:20.891270 (+ 2578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:20.891272 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:20.891272 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:59:20.891278 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:20.893166 (+ 1888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:20.893606 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:20.893607 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:20.893607 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:20.893664 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:20.893873 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:20.893874 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:20.893874 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:20.893995 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:20.894047 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:20.894278 (+ 231us) server_negotiation.cc:299] Negotiation successful
0924 04:59:20.894380 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 04:59:20.894486 7321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:20.883697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44933 (local address 127.0.98.60:44097)
0924 04:59:20.883743 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:20.883745 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:20.883816 (+ 71us) server_negotiation.cc:407] Connection header received
0924 04:59:20.884821 (+ 1005us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:20.884823 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:20.884833 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:20.884887 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:20.886120 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.886847 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.887544 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.887821 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.887884 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:20.890172 (+ 2288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:20.890174 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:20.890175 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:59:20.890183 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:20.891798 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:20.893617 (+ 1819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:20.893618 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:20.893619 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:20.893667 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:20.893978 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:20.893979 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:20.893980 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:20.894109 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:20.894155 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:20.894332 (+ 177us) server_negotiation.cc:299] Negotiation successful
0924 04:59:20.894398 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 04:59:20.894536 7727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:20.883345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43351 (local address 127.0.98.3:42405)
0924 04:59:20.883524 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:20.883563 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:20.884750 (+ 1187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:20.884936 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:20.884937 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:20.885222 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:20.885467 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:20.885473 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.886121 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.886121 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:20.887323 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:20.887325 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.887764 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.887765 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:20.887897 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:20.888354 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:20.888364 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:20.890916 (+ 2552us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:59:20.892936 (+ 2020us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:20.892937 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:20.892937 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:20.893240 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:20.893465 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:20.893465 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:20.893466 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:20.893596 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:20.893883 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:59:20.893888 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:59:20.894042 (+ 154us) client_negotiation.cc:769] Sending connection context
0924 04:59:20.894137 (+ 95us) client_negotiation.cc:240] Negotiation successful
0924 04:59:20.894315 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"mutex_wait_us":1,"thread_start_us":99,"threads_started":1}
I20260924 04:59:20.894949 7729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:20.883596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44097 (local address 127.0.98.3:44933)
0924 04:59:20.883764 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:20.883773 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:20.884609 (+ 836us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:20.884902 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:20.884903 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:20.885225 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:20.885467 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:20.885473 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.886916 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.886917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:20.887470 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:20.887471 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.887892 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.887893 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:20.887958 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:20.888470 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:20.888475 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:20.890092 (+ 1617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:59:20.892017 (+ 1925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:20.892023 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:20.892025 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:20.893535 (+ 1510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:20.893730 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:20.893731 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:20.893731 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:20.893911 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:20.894184 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:59:20.894185 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:59:20.894227 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 04:59:20.894744 (+ 517us) client_negotiation.cc:240] Negotiation successful
0924 04:59:20.894837 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":18,"thread_start_us":59,"threads_started":1}
I20260924 04:59:20.895169 7728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:20.883346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40695 (local address 127.0.98.3:55805)
0924 04:59:20.883684 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:20.883693 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:20.884595 (+ 902us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:20.884843 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:20.884847 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:20.885223 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:20.885467 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:20.885473 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.887645 (+ 2172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.887645 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:20.888277 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:20.888279 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.888705 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.888706 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:20.888779 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:20.889129 (+ 350us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:20.889134 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:20.891181 (+ 2047us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:59:20.893229 (+ 2048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:20.893230 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:20.893230 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:20.893532 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:20.893729 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:20.893729 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:20.893730 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:20.893813 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:20.894070 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:59:20.894070 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:59:20.894108 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 04:59:20.894159 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 04:59:20.894929 (+ 770us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"mutex_wait_us":663,"thread_start_us":93,"threads_started":1}
I20260924 04:59:20.895411 7723 heartbeater.cc:347] Connected to a master server at 127.0.98.60:44097
I20260924 04:59:20.895498 7723 heartbeater.cc:464] Registering TS with master...
I20260924 04:59:20.895566 6527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:20.883347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42405 (local address 127.0.98.62:43351)
0924 04:59:20.883398 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:20.883400 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:20.883642 (+ 242us) server_negotiation.cc:407] Connection header received
0924 04:59:20.884830 (+ 1188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:20.884831 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:20.884841 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:20.884889 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:20.885552 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.886027 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.887407 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.887682 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.887749 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:20.890989 (+ 3240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:20.890990 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:20.890991 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:59:20.890998 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:20.892867 (+ 1869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:20.893319 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:20.893320 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:20.893321 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:20.893380 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:20.893666 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:20.893667 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:20.893668 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:20.893799 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:20.893857 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:20.895409 (+ 1552us) server_negotiation.cc:299] Negotiation successful
0924 04:59:20.895479 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 04:59:20.895656 7723 heartbeater.cc:511] Master 127.0.98.60:44097 requested a full tablet report, sending...
I20260924 04:59:20.895677 7725 heartbeater.cc:347] Connected to a master server at 127.0.98.61:40695
I20260924 04:59:20.895714 7725 heartbeater.cc:464] Registering TS with master...
I20260924 04:59:20.895802 7725 heartbeater.cc:511] Master 127.0.98.61:40695 requested a full tablet report, sending...
I20260924 04:59:20.895956 7724 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43351
I20260924 04:59:20.895999 7724 heartbeater.cc:464] Registering TS with master...
I20260924 04:59:20.896047 7724 heartbeater.cc:511] Master 127.0.98.62:43351 requested a full tablet report, sending...
I20260924 04:59:20.896186 7084 ts_manager.cc:194] Registered new tserver with Master: b08a35b7530a47f48a8a4cf653551a6a (127.0.98.3:44049)
I20260924 04:59:20.896519 7270 ts_manager.cc:194] Registered new tserver with Master: b08a35b7530a47f48a8a4cf653551a6a (127.0.98.3:44049)
I20260924 04:59:20.896697 6487 ts_manager.cc:194] Re-registered known tserver with Master: b08a35b7530a47f48a8a4cf653551a6a (127.0.98.3:44049)
I20260924 04:59:20.897171 6487 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42405
I20260924 04:59:20.901892 7321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:20.892541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47292 (local address 127.0.98.60:44097)
0924 04:59:20.894542 (+ 2001us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:20.894544 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:20.894551 (+ 7us) server_negotiation.cc:407] Connection header received
0924 04:59:20.894567 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:20.894567 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:20.894576 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:20.894636 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:20.896009 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.896662 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.897670 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.897946 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.897986 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:20.899067 (+ 1081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:20.899068 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:20.899069 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:59:20.899074 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:20.900658 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:20.901155 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:20.901156 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:20.901157 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:20.901210 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:20.901382 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:20.901383 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:20.901384 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:20.901529 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:20.901619 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:20.901733 (+ 114us) server_negotiation.cc:299] Negotiation successful
0924 04:59:20.901797 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1987,"thread_start_us":59,"threads_started":1}
I20260924 04:59:20.902150 7736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:20.892171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45726 (local address 127.0.98.61:40695)
0924 04:59:20.892287 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:20.892288 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:20.893038 (+ 750us) server_negotiation.cc:407] Connection header received
0924 04:59:20.893309 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:20.893311 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:20.893322 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:20.893358 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:20.894074 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.894788 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.897474 (+ 2686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.897679 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.897731 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:20.899048 (+ 1317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:20.899049 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:20.899050 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:59:20.899056 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:20.900636 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:20.901202 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:20.901203 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:20.901204 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:20.901278 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:20.901477 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:20.901478 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:20.901478 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:20.901595 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:20.901674 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:20.901981 (+ 307us) server_negotiation.cc:299] Negotiation successful
0924 04:59:20.902062 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":50,"threads_started":1}
I20260924 04:59:20.903825 7734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:20.891709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34352 (local address 127.0.98.62:43351)
0924 04:59:20.891943 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:20.891945 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:20.892107 (+ 162us) server_negotiation.cc:407] Connection header received
0924 04:59:20.892134 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:20.892135 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:20.892150 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:20.892229 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:20.893183 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.898420 (+ 5237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.899616 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.899848 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.899912 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:20.901687 (+ 1775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:20.901688 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:20.901688 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:59:20.901695 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:20.903116 (+ 1421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:20.903366 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:20.903366 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:20.903366 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:20.903408 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:20.903510 (+ 102us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:20.903511 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:20.903511 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:20.903592 (+ 81us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:20.903652 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:20.903711 (+ 59us) server_negotiation.cc:299] Negotiation successful
0924 04:59:20.903767 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":70,"threads_started":1}
I20260924 04:59:20.905248 6487 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34352:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:59:20.906803 6487 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:59:20.913673 7746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:20.910026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46631 (local address 127.0.0.1:50132)
0924 04:59:20.910500 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:20.910509 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:20.910557 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:20.910719 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:20.910720 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:20.910722 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:59:20.910760 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:20.910761 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.911682 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.911683 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:20.912508 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:20.912510 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.913401 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.913402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:20.913511 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:20.913516 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 04:59:20.913560 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 04:59:20.913598 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":74,"threads_started":1}
I20260924 04:59:20.913964 7745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:20.910056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42241 (local address 127.0.0.1:33622)
0924 04:59:20.910865 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:20.910874 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:20.910922 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:20.911073 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:20.911073 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:20.911075 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:59:20.911113 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:20.911114 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.912235 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.912236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:20.913014 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:20.913016 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.913813 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.913814 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:20.913857 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:20.913860 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 04:59:20.913887 (+ 27us) client_negotiation.cc:240] Negotiation successful
0924 04:59:20.913919 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":797,"mutex_wait_us":2,"thread_start_us":83,"threads_started":1}
I20260924 04:59:20.914108 7749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:20.910220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50132 (local address 127.0.98.1:46631)
0924 04:59:20.910348 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:20.910353 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:20.910556 (+ 203us) server_negotiation.cc:407] Connection header received
0924 04:59:20.910620 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:20.910622 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:20.910640 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:20.910711 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:59:20.910808 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.911616 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.912584 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.913321 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.913395 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:20.913949 (+ 554us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:20.913983 (+ 34us) server_negotiation.cc:299] Negotiation successful
0924 04:59:20.914029 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":63,"threads_started":1}
I20260924 04:59:20.914247 7747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:20.909999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44049 (local address 127.0.0.1:57216)
0924 04:59:20.910178 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:20.910189 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:20.910276 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:20.910800 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:20.910801 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:20.910802 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:59:20.910829 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:20.910830 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.912066 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.912067 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:20.913128 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:20.913130 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.914051 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.914052 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:20.914113 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:20.914118 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 04:59:20.914145 (+ 27us) client_negotiation.cc:240] Negotiation successful
0924 04:59:20.914182 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":36,"threads_started":1}
I20260924 04:59:20.914312 7750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:20.910267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33622 (local address 127.0.98.2:42241)
0924 04:59:20.910450 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:20.910454 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:20.910927 (+ 473us) server_negotiation.cc:407] Connection header received
0924 04:59:20.910976 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:20.910978 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:20.910993 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:20.911068 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:59:20.911163 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.912163 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.913082 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.913681 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.914081 (+ 400us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:20.914167 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:20.914193 (+ 26us) server_negotiation.cc:299] Negotiation successful
0924 04:59:20.914234 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":87,"threads_started":1}
I20260924 04:59:20.914588 7748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:20.910133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57216 (local address 127.0.98.3:44049)
0924 04:59:20.910664 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:20.910669 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:20.910676 (+ 7us) server_negotiation.cc:407] Connection header received
0924 04:59:20.910711 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:20.910713 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:20.910726 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:20.910797 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:59:20.910884 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.911972 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.913221 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:20.913953 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:20.914345 (+ 392us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:20.914439 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:20.914464 (+ 25us) server_negotiation.cc:299] Negotiation successful
0924 04:59:20.914513 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":501,"thread_start_us":67,"threads_started":1}
I20260924 04:59:20.914860 7385 tablet_service.cc:1467] Processing CreateTablet for tablet 38ae5df0a6d7452fa69baecfc0495879 (DEFAULT_TABLE table=test-table [id=df4d14324e454bdb973e48d92e6430c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:59:20.914906 7521 tablet_service.cc:1467] Processing CreateTablet for tablet 38ae5df0a6d7452fa69baecfc0495879 (DEFAULT_TABLE table=test-table [id=df4d14324e454bdb973e48d92e6430c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:59:20.915218 7521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38ae5df0a6d7452fa69baecfc0495879. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:59:20.915220 7385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38ae5df0a6d7452fa69baecfc0495879. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:59:20.915166 7657 tablet_service.cc:1467] Processing CreateTablet for tablet 38ae5df0a6d7452fa69baecfc0495879 (DEFAULT_TABLE table=test-table [id=df4d14324e454bdb973e48d92e6430c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:59:20.915442 7657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38ae5df0a6d7452fa69baecfc0495879. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:59:20.918515 7752 tablet_bootstrap.cc:492] T 38ae5df0a6d7452fa69baecfc0495879 P b08a35b7530a47f48a8a4cf653551a6a: Bootstrap starting.
I20260924 04:59:20.918749 7753 tablet_bootstrap.cc:492] T 38ae5df0a6d7452fa69baecfc0495879 P 2b2dd144b90544c5bdbefde04c6672d5: Bootstrap starting.
I20260924 04:59:20.918875 7751 tablet_bootstrap.cc:492] T 38ae5df0a6d7452fa69baecfc0495879 P fc45fc89660947038800b5a9922ea04c: Bootstrap starting.
I20260924 04:59:20.919117 7752 tablet_bootstrap.cc:654] T 38ae5df0a6d7452fa69baecfc0495879 P b08a35b7530a47f48a8a4cf653551a6a: Neither blocks nor log segments found. Creating new log.
I20260924 04:59:20.919366 7752 log.cc:824] T 38ae5df0a6d7452fa69baecfc0495879 P b08a35b7530a47f48a8a4cf653551a6a: Log is configured to *not* fsync() on all Append() calls
I20260924 04:59:20.919446 7753 tablet_bootstrap.cc:654] T 38ae5df0a6d7452fa69baecfc0495879 P 2b2dd144b90544c5bdbefde04c6672d5: Neither blocks nor log segments found. Creating new log.
I20260924 04:59:20.919771 7753 log.cc:824] T 38ae5df0a6d7452fa69baecfc0495879 P 2b2dd144b90544c5bdbefde04c6672d5: Log is configured to *not* fsync() on all Append() calls
I20260924 04:59:20.919909 7751 tablet_bootstrap.cc:654] T 38ae5df0a6d7452fa69baecfc0495879 P fc45fc89660947038800b5a9922ea04c: Neither blocks nor log segments found. Creating new log.
I20260924 04:59:20.920145 7751 log.cc:824] T 38ae5df0a6d7452fa69baecfc0495879 P fc45fc89660947038800b5a9922ea04c: Log is configured to *not* fsync() on all Append() calls
I20260924 04:59:20.920490 7753 tablet_bootstrap.cc:492] T 38ae5df0a6d7452fa69baecfc0495879 P 2b2dd144b90544c5bdbefde04c6672d5: No bootstrap required, opened a new log
I20260924 04:59:20.920555 7753 ts_tablet_manager.cc:1397] T 38ae5df0a6d7452fa69baecfc0495879 P 2b2dd144b90544c5bdbefde04c6672d5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 04:59:20.920825 7751 tablet_bootstrap.cc:492] T 38ae5df0a6d7452fa69baecfc0495879 P fc45fc89660947038800b5a9922ea04c: No bootstrap required, opened a new log
I20260924 04:59:20.920882 7751 ts_tablet_manager.cc:1397] T 38ae5df0a6d7452fa69baecfc0495879 P fc45fc89660947038800b5a9922ea04c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 04:59:20.920902 7752 tablet_bootstrap.cc:492] T 38ae5df0a6d7452fa69baecfc0495879 P b08a35b7530a47f48a8a4cf653551a6a: No bootstrap required, opened a new log
I20260924 04:59:20.920953 7752 ts_tablet_manager.cc:1397] T 38ae5df0a6d7452fa69baecfc0495879 P b08a35b7530a47f48a8a4cf653551a6a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 04:59:20.922369 7753 raft_consensus.cc:348] T 38ae5df0a6d7452fa69baecfc0495879 P 2b2dd144b90544c5bdbefde04c6672d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b08a35b7530a47f48a8a4cf653551a6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44049 } } peers { permanent_uuid: "fc45fc89660947038800b5a9922ea04c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42241 } } peers { permanent_uuid: "2b2dd144b90544c5bdbefde04c6672d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46631 } }
I20260924 04:59:20.922545 7753 raft_consensus.cc:374] T 38ae5df0a6d7452fa69baecfc0495879 P 2b2dd144b90544c5bdbefde04c6672d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:59:20.922580 7753 raft_consensus.cc:729] T 38ae5df0a6d7452fa69baecfc0495879 P 2b2dd144b90544c5bdbefde04c6672d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b2dd144b90544c5bdbefde04c6672d5, State: Initialized, Role: FOLLOWER
I20260924 04:59:20.922590 7751 raft_consensus.cc:348] T 38ae5df0a6d7452fa69baecfc0495879 P fc45fc89660947038800b5a9922ea04c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b08a35b7530a47f48a8a4cf653551a6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44049 } } peers { permanent_uuid: "fc45fc89660947038800b5a9922ea04c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42241 } } peers { permanent_uuid: "2b2dd144b90544c5bdbefde04c6672d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46631 } }
I20260924 04:59:20.922739 7751 raft_consensus.cc:374] T 38ae5df0a6d7452fa69baecfc0495879 P fc45fc89660947038800b5a9922ea04c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:59:20.922711 7753 consensus_queue.cc:260] T 38ae5df0a6d7452fa69baecfc0495879 P 2b2dd144b90544c5bdbefde04c6672d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b08a35b7530a47f48a8a4cf653551a6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44049 } } peers { permanent_uuid: "fc45fc89660947038800b5a9922ea04c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42241 } } peers { permanent_uuid: "2b2dd144b90544c5bdbefde04c6672d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46631 } }
I20260924 04:59:20.922775 7751 raft_consensus.cc:729] T 38ae5df0a6d7452fa69baecfc0495879 P fc45fc89660947038800b5a9922ea04c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc45fc89660947038800b5a9922ea04c, State: Initialized, Role: FOLLOWER
I20260924 04:59:20.922875 7751 consensus_queue.cc:260] T 38ae5df0a6d7452fa69baecfc0495879 P fc45fc89660947038800b5a9922ea04c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b08a35b7530a47f48a8a4cf653551a6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44049 } } peers { permanent_uuid: "fc45fc89660947038800b5a9922ea04c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42241 } } peers { permanent_uuid: "2b2dd144b90544c5bdbefde04c6672d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46631 } }
I20260924 04:59:20.922948 7753 ts_tablet_manager.cc:1428] T 38ae5df0a6d7452fa69baecfc0495879 P 2b2dd144b90544c5bdbefde04c6672d5: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 04:59:20.923101 7751 ts_tablet_manager.cc:1428] T 38ae5df0a6d7452fa69baecfc0495879 P fc45fc89660947038800b5a9922ea04c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 04:59:20.923120 7588 heartbeater.cc:503] Master 127.0.98.62:43351 was elected leader, sending a full tablet report...
I20260924 04:59:20.923197 7752 raft_consensus.cc:348] T 38ae5df0a6d7452fa69baecfc0495879 P b08a35b7530a47f48a8a4cf653551a6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b08a35b7530a47f48a8a4cf653551a6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44049 } } peers { permanent_uuid: "fc45fc89660947038800b5a9922ea04c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42241 } } peers { permanent_uuid: "2b2dd144b90544c5bdbefde04c6672d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46631 } }
I20260924 04:59:20.923327 7752 raft_consensus.cc:374] T 38ae5df0a6d7452fa69baecfc0495879 P b08a35b7530a47f48a8a4cf653551a6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:59:20.923377 7752 raft_consensus.cc:729] T 38ae5df0a6d7452fa69baecfc0495879 P b08a35b7530a47f48a8a4cf653551a6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b08a35b7530a47f48a8a4cf653551a6a, State: Initialized, Role: FOLLOWER
I20260924 04:59:20.923478 7752 consensus_queue.cc:260] T 38ae5df0a6d7452fa69baecfc0495879 P b08a35b7530a47f48a8a4cf653551a6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b08a35b7530a47f48a8a4cf653551a6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44049 } } peers { permanent_uuid: "fc45fc89660947038800b5a9922ea04c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42241 } } peers { permanent_uuid: "2b2dd144b90544c5bdbefde04c6672d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46631 } }
I20260924 04:59:20.924031 7452 heartbeater.cc:503] Master 127.0.98.62:43351 was elected leader, sending a full tablet report...
I20260924 04:59:20.923754 7724 heartbeater.cc:503] Master 127.0.98.62:43351 was elected leader, sending a full tablet report...
I20260924 04:59:20.923699 7752 ts_tablet_manager.cc:1428] T 38ae5df0a6d7452fa69baecfc0495879 P b08a35b7530a47f48a8a4cf653551a6a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 04:59:21.001947 7321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:20.999583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47300 (local address 127.0.98.60:44097)
0924 04:59:20.999642 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:20.999644 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:20.999654 (+ 10us) server_negotiation.cc:407] Connection header received
0924 04:59:20.999700 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:20.999701 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:20.999721 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:20.999775 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:59:20.999901 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:21.000366 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:21.001228 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:21.001641 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:21.001703 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:21.001788 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:21.001826 (+ 38us) server_negotiation.cc:299] Negotiation successful
0924 04:59:21.001861 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 04:59:21.001947 7745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:20.999527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44097 (local address 127.0.0.1:47300)
0924 04:59:20.999581 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:20.999592 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:20.999644 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:20.999783 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:20.999784 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:20.999785 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:59:20.999836 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:20.999839 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:21.000434 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:21.000436 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:21.001142 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:21.001144 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:21.001711 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:21.001712 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:21.001771 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:21.001776 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 04:59:21.001821 (+ 45us) client_negotiation.cc:240] Negotiation successful
0924 04:59:21.001868 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260924 04:59:21.002354 7759 raft_consensus.cc:482] T 38ae5df0a6d7452fa69baecfc0495879 P b08a35b7530a47f48a8a4cf653551a6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:59:21.002415 7280 raft_consensus.cc:1203] T 00000000000000000000000000000000 P a65aed7f2fdc4e6ea4e8765bbe365a42 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.9] Dedup: 1.7->[1.8-1.9]
I20260924 04:59:21.002468 7759 raft_consensus.cc:504] T 38ae5df0a6d7452fa69baecfc0495879 P b08a35b7530a47f48a8a4cf653551a6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b08a35b7530a47f48a8a4cf653551a6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44049 } } peers { permanent_uuid: "fc45fc89660947038800b5a9922ea04c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42241 } } peers { permanent_uuid: "2b2dd144b90544c5bdbefde04c6672d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46631 } }
I20260924 04:59:21.002802 7759 leader_election.cc:288] T 38ae5df0a6d7452fa69baecfc0495879 P b08a35b7530a47f48a8a4cf653551a6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fc45fc89660947038800b5a9922ea04c (127.0.98.2:42241), 2b2dd144b90544c5bdbefde04c6672d5 (127.0.98.1:46631)
I20260924 04:59:21.003401 7760 sys_catalog.cc:455] T 00000000000000000000000000000000 P a65aed7f2fdc4e6ea4e8765bbe365a42 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 221d34b534b046078e9003aff8289b32. Latest consensus state: current_term: 1 leader_uuid: "221d34b534b046078e9003aff8289b32" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "221d34b534b046078e9003aff8289b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43351 } } peers { permanent_uuid: "c82275d62aac4bc580771ab80ac00860" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40695 } attrs { promote: false } } peers { permanent_uuid: "a65aed7f2fdc4e6ea4e8765bbe365a42" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44097 } attrs { promote: true } } }
I20260924 04:59:21.003515 7760 sys_catalog.cc:458] T 00000000000000000000000000000000 P a65aed7f2fdc4e6ea4e8765bbe365a42 [sys.catalog]: This master's current role is: LEARNER
I20260924 04:59:21.005730 7749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:21.003137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40269 (local address 127.0.98.1:46631)
0924 04:59:21.003188 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:21.003191 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:21.003201 (+ 10us) server_negotiation.cc:407] Connection header received
0924 04:59:21.003228 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:21.003229 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:21.003239 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:21.003293 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:59:21.003479 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:21.004115 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:21.005019 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:21.005435 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:21.005492 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:21.005502 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:21.005629 (+ 127us) server_negotiation.cc:299] Negotiation successful
0924 04:59:21.005669 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 04:59:21.005743 7728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:21.002949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46631 (local address 127.0.98.3:40269)
0924 04:59:21.003057 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:21.003066 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:21.003095 (+ 29us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:21.003344 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:21.003345 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:21.003346 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:59:21.003405 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:21.003406 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:21.004178 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:21.004179 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:21.004945 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:21.004947 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:21.005496 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:21.005497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:21.005572 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:21.005575 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 04:59:21.005619 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 04:59:21.005656 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"mutex_wait_us":37}
I20260924 04:59:21.005767 7750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:21.003068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57851 (local address 127.0.98.2:42241)
0924 04:59:21.003133 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:21.003135 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:21.003141 (+ 6us) server_negotiation.cc:407] Connection header received
0924 04:59:21.003164 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:21.003166 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:21.003176 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:21.003216 (+ 40us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:59:21.003386 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:21.004017 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:21.004972 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:21.005450 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:21.005511 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:21.005529 (+ 18us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:21.005661 (+ 132us) server_negotiation.cc:299] Negotiation successful
0924 04:59:21.005694 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 04:59:21.006134 7729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:21.002948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42241 (local address 127.0.98.3:57851)
0924 04:59:21.003020 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:21.003029 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:21.003063 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:21.003254 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:21.003255 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:21.003256 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:59:21.003313 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:21.003315 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:21.004122 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:21.004123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:21.004892 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:21.004895 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:21.005510 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:21.005511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:21.005603 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:21.005606 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 04:59:21.005650 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 04:59:21.005689 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260924 04:59:21.006251 7405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38ae5df0a6d7452fa69baecfc0495879" candidate_uuid: "b08a35b7530a47f48a8a4cf653551a6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b2dd144b90544c5bdbefde04c6672d5" is_pre_election: true
I20260924 04:59:21.006412 7405 raft_consensus.cc:2454] T 38ae5df0a6d7452fa69baecfc0495879 P 2b2dd144b90544c5bdbefde04c6672d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b08a35b7530a47f48a8a4cf653551a6a in term 0.
I20260924 04:59:21.006470 7541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38ae5df0a6d7452fa69baecfc0495879" candidate_uuid: "b08a35b7530a47f48a8a4cf653551a6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc45fc89660947038800b5a9922ea04c" is_pre_election: true
I20260924 04:59:21.006570 7541 raft_consensus.cc:2454] T 38ae5df0a6d7452fa69baecfc0495879 P fc45fc89660947038800b5a9922ea04c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b08a35b7530a47f48a8a4cf653551a6a in term 0.
I20260924 04:59:21.006589 7613 leader_election.cc:302] T 38ae5df0a6d7452fa69baecfc0495879 P b08a35b7530a47f48a8a4cf653551a6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b2dd144b90544c5bdbefde04c6672d5, b08a35b7530a47f48a8a4cf653551a6a; no voters:
I20260924 04:59:21.006693 7759 raft_consensus.cc:2781] T 38ae5df0a6d7452fa69baecfc0495879 P b08a35b7530a47f48a8a4cf653551a6a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:59:21.006739 7759 raft_consensus.cc:482] T 38ae5df0a6d7452fa69baecfc0495879 P b08a35b7530a47f48a8a4cf653551a6a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:59:21.006758 7759 raft_consensus.cc:3037] T 38ae5df0a6d7452fa69baecfc0495879 P b08a35b7530a47f48a8a4cf653551a6a [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:59:21.007400 7759 raft_consensus.cc:504] T 38ae5df0a6d7452fa69baecfc0495879 P b08a35b7530a47f48a8a4cf653551a6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b08a35b7530a47f48a8a4cf653551a6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44049 } } peers { permanent_uuid: "fc45fc89660947038800b5a9922ea04c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42241 } } peers { permanent_uuid: "2b2dd144b90544c5bdbefde04c6672d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46631 } }
I20260924 04:59:21.007510 7759 leader_election.cc:288] T 38ae5df0a6d7452fa69baecfc0495879 P b08a35b7530a47f48a8a4cf653551a6a [CANDIDATE]: Term 1 election: Requested vote from peers fc45fc89660947038800b5a9922ea04c (127.0.98.2:42241), 2b2dd144b90544c5bdbefde04c6672d5 (127.0.98.1:46631)
W20260924 04:59:21.007604 7590 tablet.cc:2369] T 38ae5df0a6d7452fa69baecfc0495879 P fc45fc89660947038800b5a9922ea04c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:59:21.007647 7541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38ae5df0a6d7452fa69baecfc0495879" candidate_uuid: "b08a35b7530a47f48a8a4cf653551a6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc45fc89660947038800b5a9922ea04c"
I20260924 04:59:21.007727 7541 raft_consensus.cc:3037] T 38ae5df0a6d7452fa69baecfc0495879 P fc45fc89660947038800b5a9922ea04c [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:59:21.008404 7541 raft_consensus.cc:2454] T 38ae5df0a6d7452fa69baecfc0495879 P fc45fc89660947038800b5a9922ea04c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b08a35b7530a47f48a8a4cf653551a6a in term 1.
I20260924 04:59:21.008562 7612 leader_election.cc:302] T 38ae5df0a6d7452fa69baecfc0495879 P b08a35b7530a47f48a8a4cf653551a6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b08a35b7530a47f48a8a4cf653551a6a, fc45fc89660947038800b5a9922ea04c; no voters:
I20260924 04:59:21.008661 7405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38ae5df0a6d7452fa69baecfc0495879" candidate_uuid: "b08a35b7530a47f48a8a4cf653551a6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b2dd144b90544c5bdbefde04c6672d5"
I20260924 04:59:21.008728 7405 raft_consensus.cc:3037] T 38ae5df0a6d7452fa69baecfc0495879 P 2b2dd144b90544c5bdbefde04c6672d5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:59:21.009496 7759 raft_consensus.cc:2781] T 38ae5df0a6d7452fa69baecfc0495879 P b08a35b7530a47f48a8a4cf653551a6a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:59:21.009580 7405 raft_consensus.cc:2454] T 38ae5df0a6d7452fa69baecfc0495879 P 2b2dd144b90544c5bdbefde04c6672d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b08a35b7530a47f48a8a4cf653551a6a in term 1.
I20260924 04:59:21.009646 7759 raft_consensus.cc:686] T 38ae5df0a6d7452fa69baecfc0495879 P b08a35b7530a47f48a8a4cf653551a6a [term 1 LEADER]: Becoming Leader. State: Replica: b08a35b7530a47f48a8a4cf653551a6a, State: Running, Role: LEADER
I20260924 04:59:21.009738 7759 consensus_queue.cc:237] T 38ae5df0a6d7452fa69baecfc0495879 P b08a35b7530a47f48a8a4cf653551a6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b08a35b7530a47f48a8a4cf653551a6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44049 } } peers { permanent_uuid: "fc45fc89660947038800b5a9922ea04c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42241 } } peers { permanent_uuid: "2b2dd144b90544c5bdbefde04c6672d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46631 } }
I20260924 04:59:21.010571 6486 catalog_manager.cc:5495] T 38ae5df0a6d7452fa69baecfc0495879 P b08a35b7530a47f48a8a4cf653551a6a reported cstate change: term changed from 0 to 1, leader changed from <none> to b08a35b7530a47f48a8a4cf653551a6a (127.0.98.3). New cstate: current_term: 1 leader_uuid: "b08a35b7530a47f48a8a4cf653551a6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b08a35b7530a47f48a8a4cf653551a6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fc45fc89660947038800b5a9922ea04c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42241 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b2dd144b90544c5bdbefde04c6672d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46631 } health_report { overall_health: UNKNOWN } } }
I20260924 04:59:21.011135 7741 sys_catalog.cc:455] T 00000000000000000000000000000000 P 221d34b534b046078e9003aff8289b32 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "221d34b534b046078e9003aff8289b32" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "221d34b534b046078e9003aff8289b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43351 } } peers { permanent_uuid: "c82275d62aac4bc580771ab80ac00860" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40695 } attrs { promote: false } } peers { permanent_uuid: "a65aed7f2fdc4e6ea4e8765bbe365a42" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44097 } attrs { promote: true } } }
I20260924 04:59:21.011224 7741 sys_catalog.cc:458] T 00000000000000000000000000000000 P 221d34b534b046078e9003aff8289b32 [sys.catalog]: This master's current role is: LEADER
I20260924 04:59:21.011763 7741 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 221d34b534b046078e9003aff8289b32: attempting to promote NON_VOTER a65aed7f2fdc4e6ea4e8765bbe365a42 to VOTER
I20260924 04:59:21.011890 7741 consensus_queue.cc:237] T 00000000000000000000000000000000 P 221d34b534b046078e9003aff8289b32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "221d34b534b046078e9003aff8289b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43351 } } peers { permanent_uuid: "c82275d62aac4bc580771ab80ac00860" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40695 } attrs { promote: false } } peers { permanent_uuid: "a65aed7f2fdc4e6ea4e8765bbe365a42" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44097 } attrs { promote: false } }
I20260924 04:59:21.012411 7280 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a65aed7f2fdc4e6ea4e8765bbe365a42 [term 1 LEARNER]: Refusing update from remote peer 221d34b534b046078e9003aff8289b32: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260924 04:59:21.012421 7094 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c82275d62aac4bc580771ab80ac00860 [term 1 FOLLOWER]: Refusing update from remote peer 221d34b534b046078e9003aff8289b32: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260924 04:59:21.012586 7767 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 221d34b534b046078e9003aff8289b32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a65aed7f2fdc4e6ea4e8765bbe365a42" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44097 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260924 04:59:21.012687 7768 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 221d34b534b046078e9003aff8289b32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c82275d62aac4bc580771ab80ac00860" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40695 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260924 04:59:21.013221 7741 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 221d34b534b046078e9003aff8289b32 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 7 to 11, a65aed7f2fdc4e6ea4e8765bbe365a42 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "221d34b534b046078e9003aff8289b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43351 } } peers { permanent_uuid: "c82275d62aac4bc580771ab80ac00860" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40695 } attrs { promote: false } } peers { permanent_uuid: "a65aed7f2fdc4e6ea4e8765bbe365a42" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44097 } attrs { promote: false } } }
I20260924 04:59:21.013924 7741 sys_catalog.cc:455] T 00000000000000000000000000000000 P 221d34b534b046078e9003aff8289b32 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "221d34b534b046078e9003aff8289b32" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "221d34b534b046078e9003aff8289b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43351 } } peers { permanent_uuid: "c82275d62aac4bc580771ab80ac00860" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40695 } attrs { promote: false } } peers { permanent_uuid: "a65aed7f2fdc4e6ea4e8765bbe365a42" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44097 } attrs { promote: false } } }
I20260924 04:59:21.013998 7741 sys_catalog.cc:458] T 00000000000000000000000000000000 P 221d34b534b046078e9003aff8289b32 [sys.catalog]: This master's current role is: LEADER
I20260924 04:59:21.013976 7231 sys_catalog.cc:455] T 00000000000000000000000000000000 P 221d34b534b046078e9003aff8289b32 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "221d34b534b046078e9003aff8289b32" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "221d34b534b046078e9003aff8289b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43351 } } peers { permanent_uuid: "c82275d62aac4bc580771ab80ac00860" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40695 } attrs { promote: false } } peers { permanent_uuid: "a65aed7f2fdc4e6ea4e8765bbe365a42" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44097 } attrs { promote: false } } }
I20260924 04:59:21.013952 7767 sys_catalog.cc:455] T 00000000000000000000000000000000 P 221d34b534b046078e9003aff8289b32 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "221d34b534b046078e9003aff8289b32" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "221d34b534b046078e9003aff8289b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43351 } } peers { permanent_uuid: "c82275d62aac4bc580771ab80ac00860" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40695 } attrs { promote: false } } peers { permanent_uuid: "a65aed7f2fdc4e6ea4e8765bbe365a42" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44097 } attrs { promote: false } } }
I20260924 04:59:21.014045 7231 sys_catalog.cc:458] T 00000000000000000000000000000000 P 221d34b534b046078e9003aff8289b32 [sys.catalog]: This master's current role is: LEADER
I20260924 04:59:21.014052 7767 sys_catalog.cc:458] T 00000000000000000000000000000000 P 221d34b534b046078e9003aff8289b32 [sys.catalog]: This master's current role is: LEADER
I20260924 04:59:21.014163 7094 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c82275d62aac4bc580771ab80ac00860 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, a65aed7f2fdc4e6ea4e8765bbe365a42 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "221d34b534b046078e9003aff8289b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43351 } } peers { permanent_uuid: "c82275d62aac4bc580771ab80ac00860" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40695 } attrs { promote: false } } peers { permanent_uuid: "a65aed7f2fdc4e6ea4e8765bbe365a42" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44097 } attrs { promote: false } } }
I20260924 04:59:21.014192 7280 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a65aed7f2fdc4e6ea4e8765bbe365a42 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, a65aed7f2fdc4e6ea4e8765bbe365a42 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "221d34b534b046078e9003aff8289b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43351 } } peers { permanent_uuid: "c82275d62aac4bc580771ab80ac00860" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40695 } attrs { promote: false } } peers { permanent_uuid: "a65aed7f2fdc4e6ea4e8765bbe365a42" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44097 } attrs { promote: false } } }
I20260924 04:59:21.014889 7770 sys_catalog.cc:455] T 00000000000000000000000000000000 P c82275d62aac4bc580771ab80ac00860 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "221d34b534b046078e9003aff8289b32" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "221d34b534b046078e9003aff8289b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43351 } } peers { permanent_uuid: "c82275d62aac4bc580771ab80ac00860" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40695 } attrs { promote: false } } peers { permanent_uuid: "a65aed7f2fdc4e6ea4e8765bbe365a42" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44097 } attrs { promote: false } } }
I20260924 04:59:21.014978 7770 sys_catalog.cc:458] T 00000000000000000000000000000000 P c82275d62aac4bc580771ab80ac00860 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:59:21.015022 7760 sys_catalog.cc:455] T 00000000000000000000000000000000 P a65aed7f2fdc4e6ea4e8765bbe365a42 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "221d34b534b046078e9003aff8289b32" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "221d34b534b046078e9003aff8289b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43351 } } peers { permanent_uuid: "c82275d62aac4bc580771ab80ac00860" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40695 } attrs { promote: false } } peers { permanent_uuid: "a65aed7f2fdc4e6ea4e8765bbe365a42" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44097 } attrs { promote: false } } }
I20260924 04:59:21.015102 7760 sys_catalog.cc:458] T 00000000000000000000000000000000 P a65aed7f2fdc4e6ea4e8765bbe365a42 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:59:21.030869 7748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:21.028758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57224 (local address 127.0.98.3:44049)
0924 04:59:21.028801 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:21.028803 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:21.028810 (+ 7us) server_negotiation.cc:407] Connection header received
0924 04:59:21.028862 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:21.028863 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:21.028881 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:21.028946 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:59:21.029055 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:21.029561 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:21.030255 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:21.030433 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:21.030476 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:21.030622 (+ 146us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:59:21.030720 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:59:21.030775 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:21.030789 (+ 14us) server_negotiation.cc:299] Negotiation successful
0924 04:59:21.030804 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 04:59:21.032655 7405 raft_consensus.cc:1261] T 38ae5df0a6d7452fa69baecfc0495879 P 2b2dd144b90544c5bdbefde04c6672d5 [term 1 FOLLOWER]: Refusing update from remote peer b08a35b7530a47f48a8a4cf653551a6a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:59:21.032651 7541 raft_consensus.cc:1261] T 38ae5df0a6d7452fa69baecfc0495879 P fc45fc89660947038800b5a9922ea04c [term 1 FOLLOWER]: Refusing update from remote peer b08a35b7530a47f48a8a4cf653551a6a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:59:21.032979 7759 consensus_queue.cc:1035] T 38ae5df0a6d7452fa69baecfc0495879 P b08a35b7530a47f48a8a4cf653551a6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc45fc89660947038800b5a9922ea04c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:59:21.033118 7765 consensus_queue.cc:1035] T 38ae5df0a6d7452fa69baecfc0495879 P b08a35b7530a47f48a8a4cf653551a6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b2dd144b90544c5bdbefde04c6672d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:59:21.035604 7772 mvcc.cc:205] Tried to move back new op lower bound from 7332765536387022848 to 7332765536296505344. Current Snapshot: MvccSnapshot[applied={T|T < 7332765536387022848}]
I20260924 04:59:21.035794 7774 mvcc.cc:205] Tried to move back new op lower bound from 7332765536387022848 to 7332765536296505344. Current Snapshot: MvccSnapshot[applied={T|T < 7332765536387022848}]
I20260924 04:59:21.036991 7773 mvcc.cc:205] Tried to move back new op lower bound from 7332765536387022848 to 7332765536296505344. Current Snapshot: MvccSnapshot[applied={T|T < 7332765536387022848}]
I20260924 04:59:21.037856 6487 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34352:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:59:21.037945 6487 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34352:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:59:21.039287 6487 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 221d34b534b046078e9003aff8289b32: Sending DeleteTablet for 3 replicas of tablet 38ae5df0a6d7452fa69baecfc0495879
I20260924 04:59:21.039577 7521 tablet_service.cc:1514] Processing DeleteTablet for tablet 38ae5df0a6d7452fa69baecfc0495879 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:59:21 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33622
I20260924 04:59:21.039649 7385 tablet_service.cc:1514] Processing DeleteTablet for tablet 38ae5df0a6d7452fa69baecfc0495879 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:59:21 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50132
I20260924 04:59:21.039922 7782 tablet_replica.cc:330] T 38ae5df0a6d7452fa69baecfc0495879 P 2b2dd144b90544c5bdbefde04c6672d5: stopping tablet replica
I20260924 04:59:21.039925 7657 tablet_service.cc:1514] Processing DeleteTablet for tablet 38ae5df0a6d7452fa69baecfc0495879 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:59:21 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57216
I20260924 04:59:21.040020 7782 raft_consensus.cc:2229] T 38ae5df0a6d7452fa69baecfc0495879 P 2b2dd144b90544c5bdbefde04c6672d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:59:21.040119 7782 raft_consensus.cc:2258] T 38ae5df0a6d7452fa69baecfc0495879 P 2b2dd144b90544c5bdbefde04c6672d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:59:21.040552 7782 ts_tablet_manager.cc:1905] T 38ae5df0a6d7452fa69baecfc0495879 P 2b2dd144b90544c5bdbefde04c6672d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:59:21.040624 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 6455
I20260924 04:59:21.040661 7781 tablet_replica.cc:330] T 38ae5df0a6d7452fa69baecfc0495879 P fc45fc89660947038800b5a9922ea04c: stopping tablet replica
I20260924 04:59:21.042006 7781 raft_consensus.cc:2229] T 38ae5df0a6d7452fa69baecfc0495879 P fc45fc89660947038800b5a9922ea04c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:59:21.042091 7782 ts_tablet_manager.cc:1918] T 38ae5df0a6d7452fa69baecfc0495879 P 2b2dd144b90544c5bdbefde04c6672d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:59:21.042114 7781 raft_consensus.cc:2258] T 38ae5df0a6d7452fa69baecfc0495879 P fc45fc89660947038800b5a9922ea04c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:59:21.042160 7782 log.cc:1196] T 38ae5df0a6d7452fa69baecfc0495879 P 2b2dd144b90544c5bdbefde04c6672d5: Deleting WAL directory at /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/ts-0/wal/wals/38ae5df0a6d7452fa69baecfc0495879
I20260924 04:59:21.042431 7782 ts_tablet_manager.cc:1939] T 38ae5df0a6d7452fa69baecfc0495879 P 2b2dd144b90544c5bdbefde04c6672d5: Deleting consensus metadata
I20260924 04:59:21.042609 7781 ts_tablet_manager.cc:1905] T 38ae5df0a6d7452fa69baecfc0495879 P fc45fc89660947038800b5a9922ea04c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:59:21.042783 7783 tablet_replica.cc:330] T 38ae5df0a6d7452fa69baecfc0495879 P b08a35b7530a47f48a8a4cf653551a6a: stopping tablet replica
I20260924 04:59:21.042867 7783 raft_consensus.cc:2229] T 38ae5df0a6d7452fa69baecfc0495879 P b08a35b7530a47f48a8a4cf653551a6a [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:59:21.042968 7783 raft_consensus.cc:2258] T 38ae5df0a6d7452fa69baecfc0495879 P b08a35b7530a47f48a8a4cf653551a6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:59:21.043390 7783 ts_tablet_manager.cc:1905] T 38ae5df0a6d7452fa69baecfc0495879 P b08a35b7530a47f48a8a4cf653551a6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:59:21.044232 7781 ts_tablet_manager.cc:1918] T 38ae5df0a6d7452fa69baecfc0495879 P fc45fc89660947038800b5a9922ea04c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:59:21.044292 7781 log.cc:1196] T 38ae5df0a6d7452fa69baecfc0495879 P fc45fc89660947038800b5a9922ea04c: Deleting WAL directory at /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/ts-1/wal/wals/38ae5df0a6d7452fa69baecfc0495879
I20260924 04:59:21.044541 7781 ts_tablet_manager.cc:1939] T 38ae5df0a6d7452fa69baecfc0495879 P fc45fc89660947038800b5a9922ea04c: Deleting consensus metadata
I20260924 04:59:21.044873 7783 ts_tablet_manager.cc:1918] T 38ae5df0a6d7452fa69baecfc0495879 P b08a35b7530a47f48a8a4cf653551a6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:59:21.044926 7783 log.cc:1196] T 38ae5df0a6d7452fa69baecfc0495879 P b08a35b7530a47f48a8a4cf653551a6a: Deleting WAL directory at /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/ts-2/wal/wals/38ae5df0a6d7452fa69baecfc0495879
I20260924 04:59:21.045128 7783 ts_tablet_manager.cc:1939] T 38ae5df0a6d7452fa69baecfc0495879 P b08a35b7530a47f48a8a4cf653551a6a: Deleting consensus metadata
W20260924 04:59:21.046501 7477 connection.cc:535] server connection from 127.0.0.1:33622 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260924 04:59:21.046526 7611 connection.cc:535] server connection from 127.0.0.1:57216 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260924 04:59:21.046684 7338 connection.cc:535] server connection from 127.0.0.1:50132 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 04:59:21.054890 7736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:21.047817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45728 (local address 127.0.98.61:40695)
0924 04:59:21.047857 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:21.047859 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:21.048118 (+ 259us) server_negotiation.cc:407] Connection header received
0924 04:59:21.048164 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:21.048165 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:21.048180 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:21.048244 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:21.048706 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:21.049514 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:21.050498 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:21.050711 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:21.050762 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:21.051895 (+ 1133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:21.051897 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:21.051897 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:59:21.051904 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:21.053771 (+ 1867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:21.054149 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:21.054150 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:21.054150 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:21.054218 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:21.054395 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:21.054396 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:21.054396 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:21.054526 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:21.054625 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:21.054711 (+ 86us) server_negotiation.cc:299] Negotiation successful
0924 04:59:21.054798 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 04:59:21.056977 7321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:21.047990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47302 (local address 127.0.98.60:44097)
0924 04:59:21.048035 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:21.048038 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:21.049588 (+ 1550us) server_negotiation.cc:407] Connection header received
0924 04:59:21.049711 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:21.049712 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:21.049724 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:21.049775 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:21.050419 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:21.051151 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:21.052229 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:21.052511 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:21.052561 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:21.054534 (+ 1973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:21.054535 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:21.054535 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:59:21.054541 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:21.056085 (+ 1544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:21.056369 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:21.056370 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:21.056371 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:21.056428 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:21.056556 (+ 128us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:21.056557 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:21.056557 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:21.056673 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:21.056760 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:21.056817 (+ 57us) server_negotiation.cc:299] Negotiation successful
0924 04:59:21.056887 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 04:59:22.037173 7791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:22.036926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43351 (local address 127.0.98.2:36855)
0924 04:59:22.037045 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:22.037118 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":56,"threads_started":1}
I20260924 04:59:22.037300 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:22.037063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43351 (local address 127.0.98.1:36765)
0924 04:59:22.037178 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:22.037250 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":46,"threads_started":1}
W20260924 04:59:22.037468 7588 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43351 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:43351: connect: Connection refused (error 111)
I20260924 04:59:22.037606 7793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:22.037282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43351 (local address 127.0.98.3:56545)
0924 04:59:22.037502 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:22.037565 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":43,"threads_started":1}
W20260924 04:59:22.037689 7452 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43351 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:43351: connect: Connection refused (error 111)
W20260924 04:59:22.037782 7724 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43351 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:43351: connect: Connection refused (error 111)
I20260924 04:59:22.539311 7794 raft_consensus.cc:482] T 00000000000000000000000000000000 P a65aed7f2fdc4e6ea4e8765bbe365a42 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 221d34b534b046078e9003aff8289b32)
I20260924 04:59:22.539470 7794 raft_consensus.cc:504] T 00000000000000000000000000000000 P a65aed7f2fdc4e6ea4e8765bbe365a42 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "221d34b534b046078e9003aff8289b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43351 } } peers { permanent_uuid: "c82275d62aac4bc580771ab80ac00860" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40695 } attrs { promote: false } } peers { permanent_uuid: "a65aed7f2fdc4e6ea4e8765bbe365a42" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44097 } attrs { promote: false } }
I20260924 04:59:22.539755 7794 leader_election.cc:288] T 00000000000000000000000000000000 P a65aed7f2fdc4e6ea4e8765bbe365a42 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 221d34b534b046078e9003aff8289b32 (127.0.98.62:43351), c82275d62aac4bc580771ab80ac00860 (127.0.98.61:40695)
I20260924 04:59:22.539991 7795 raft_consensus.cc:482] T 00000000000000000000000000000000 P c82275d62aac4bc580771ab80ac00860 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 221d34b534b046078e9003aff8289b32)
I20260924 04:59:22.540076 7796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:22.539866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43351 (local address 127.0.0.1:34868)
0924 04:59:22.539980 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:22.540031 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":59,"threads_started":1}
I20260924 04:59:22.540071 7795 raft_consensus.cc:504] T 00000000000000000000000000000000 P c82275d62aac4bc580771ab80ac00860 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "221d34b534b046078e9003aff8289b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43351 } } peers { permanent_uuid: "c82275d62aac4bc580771ab80ac00860" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40695 } attrs { promote: false } } peers { permanent_uuid: "a65aed7f2fdc4e6ea4e8765bbe365a42" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44097 } attrs { promote: false } }
W20260924 04:59:22.540189 7257 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43351: connect: Connection refused (error 111)
I20260924 04:59:22.540282 7795 leader_election.cc:288] T 00000000000000000000000000000000 P c82275d62aac4bc580771ab80ac00860 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 221d34b534b046078e9003aff8289b32 (127.0.98.62:43351), a65aed7f2fdc4e6ea4e8765bbe365a42 (127.0.98.60:44097)
I20260924 04:59:22.540548 7800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:22.540366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43351 (local address 127.0.0.1:34884)
0924 04:59:22.540461 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:22.540512 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82,"thread_start_us":46,"threads_started":1}
W20260924 04:59:22.540686 7071 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43351: connect: Connection refused (error 111)
I20260924 04:59:22.540838 7796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:22.540730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43351 (local address 127.0.0.1:34886)
0924 04:59:22.540776 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:22.540808 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38,"thread_start_us":53,"threads_started":1}
W20260924 04:59:22.540926 7257 leader_election.cc:334] T 00000000000000000000000000000000 P a65aed7f2fdc4e6ea4e8765bbe365a42 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 221d34b534b046078e9003aff8289b32 (127.0.98.62:43351): Network error: Client connection negotiation failed: client connection to 127.0.98.62:43351: connect: Connection refused (error 111)
I20260924 04:59:22.541250 7800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:22.541154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43351 (local address 127.0.0.1:34888)
0924 04:59:22.541198 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:22.541225 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260924 04:59:22.541333 7071 leader_election.cc:334] T 00000000000000000000000000000000 P c82275d62aac4bc580771ab80ac00860 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 221d34b534b046078e9003aff8289b32 (127.0.98.62:43351): Network error: Client connection negotiation failed: client connection to 127.0.98.62:43351: connect: Connection refused (error 111)
I20260924 04:59:22.543169 7798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:22.539971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45734 (local address 127.0.98.61:40695)
0924 04:59:22.540055 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:22.540057 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:22.540278 (+ 221us) server_negotiation.cc:407] Connection header received
0924 04:59:22.540306 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:22.540307 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:22.540318 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:22.540370 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:59:22.540586 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:22.541258 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:22.542532 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:22.542892 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:22.542952 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:22.542963 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:22.543075 (+ 112us) server_negotiation.cc:299] Negotiation successful
0924 04:59:22.543109 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61,"thread_start_us":41,"threads_started":1}
I20260924 04:59:22.543181 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:22.539932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40695 (local address 127.0.0.1:45734)
0924 04:59:22.540152 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:22.540162 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:22.540225 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:22.540456 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:22.540457 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:22.540460 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:59:22.540514 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:22.540516 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:22.541348 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:22.541348 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:22.542466 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:22.542468 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:22.542959 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:22.542960 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:22.543020 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:22.543023 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 04:59:22.543065 (+ 42us) client_negotiation.cc:240] Negotiation successful
0924 04:59:22.543105 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":43,"threads_started":1}
I20260924 04:59:22.543363 7094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a65aed7f2fdc4e6ea4e8765bbe365a42" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "c82275d62aac4bc580771ab80ac00860" is_pre_election: true
I20260924 04:59:22.543447 7094 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c82275d62aac4bc580771ab80ac00860 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a65aed7f2fdc4e6ea4e8765bbe365a42 in term 1.
I20260924 04:59:22.543596 7254 leader_election.cc:302] T 00000000000000000000000000000000 P a65aed7f2fdc4e6ea4e8765bbe365a42 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a65aed7f2fdc4e6ea4e8765bbe365a42, c82275d62aac4bc580771ab80ac00860; no voters: 221d34b534b046078e9003aff8289b32
I20260924 04:59:22.543720 7794 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a65aed7f2fdc4e6ea4e8765bbe365a42 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 04:59:22.543761 7794 raft_consensus.cc:482] T 00000000000000000000000000000000 P a65aed7f2fdc4e6ea4e8765bbe365a42 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 221d34b534b046078e9003aff8289b32)
I20260924 04:59:22.543779 7794 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a65aed7f2fdc4e6ea4e8765bbe365a42 [term 1 FOLLOWER]: Advancing to term 2
I20260924 04:59:22.544087 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:22.540585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47312 (local address 127.0.98.60:44097)
0924 04:59:22.540724 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:22.540726 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:22.540736 (+ 10us) server_negotiation.cc:407] Connection header received
0924 04:59:22.541094 (+ 358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:22.541095 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:22.541105 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:22.541156 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:59:22.541479 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:22.542200 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:22.543248 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:22.543796 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:22.543880 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:22.543890 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:22.544015 (+ 125us) server_negotiation.cc:299] Negotiation successful
0924 04:59:22.544045 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":74,"threads_started":1}
I20260924 04:59:22.544098 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:22.540410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44097 (local address 127.0.0.1:47312)
0924 04:59:22.540537 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:22.540547 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:22.541020 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:22.541228 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:22.541229 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:22.541231 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:59:22.541272 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:22.541274 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:22.542273 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:22.542274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:22.543144 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:22.543146 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:22.543885 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:22.543886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:22.543945 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:22.543949 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 04:59:22.544007 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 04:59:22.544041 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":49,"threads_started":1}
I20260924 04:59:22.544277 7280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c82275d62aac4bc580771ab80ac00860" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "a65aed7f2fdc4e6ea4e8765bbe365a42" is_pre_election: true
I20260924 04:59:22.544632 7794 raft_consensus.cc:504] T 00000000000000000000000000000000 P a65aed7f2fdc4e6ea4e8765bbe365a42 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "221d34b534b046078e9003aff8289b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43351 } } peers { permanent_uuid: "c82275d62aac4bc580771ab80ac00860" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40695 } attrs { promote: false } } peers { permanent_uuid: "a65aed7f2fdc4e6ea4e8765bbe365a42" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44097 } attrs { promote: false } }
I20260924 04:59:22.544744 7280 raft_consensus.cc:2379] T 00000000000000000000000000000000 P a65aed7f2fdc4e6ea4e8765bbe365a42 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c82275d62aac4bc580771ab80ac00860 in current term 2: Already voted for candidate a65aed7f2fdc4e6ea4e8765bbe365a42 in this term.
I20260924 04:59:22.544745 7794 leader_election.cc:288] T 00000000000000000000000000000000 P a65aed7f2fdc4e6ea4e8765bbe365a42 [CANDIDATE]: Term 2 election: Requested vote from peers 221d34b534b046078e9003aff8289b32 (127.0.98.62:43351), c82275d62aac4bc580771ab80ac00860 (127.0.98.61:40695)
I20260924 04:59:22.544893 7094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a65aed7f2fdc4e6ea4e8765bbe365a42" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "c82275d62aac4bc580771ab80ac00860"
I20260924 04:59:22.544950 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:22.544835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43351 (local address 127.0.0.1:34898)
0924 04:59:22.544901 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:22.544924 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260924 04:59:22.544957 7094 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c82275d62aac4bc580771ab80ac00860 [term 1 FOLLOWER]: Advancing to term 2
I20260924 04:59:22.544970 7068 leader_election.cc:302] T 00000000000000000000000000000000 P c82275d62aac4bc580771ab80ac00860 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c82275d62aac4bc580771ab80ac00860; no voters: 221d34b534b046078e9003aff8289b32, a65aed7f2fdc4e6ea4e8765bbe365a42
I20260924 04:59:22.545267 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:22.545179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43351 (local address 127.0.0.1:34900)
0924 04:59:22.545215 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:22.545238 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260924 04:59:22.545334 7257 leader_election.cc:334] T 00000000000000000000000000000000 P a65aed7f2fdc4e6ea4e8765bbe365a42 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 221d34b534b046078e9003aff8289b32 (127.0.98.62:43351): Network error: Client connection negotiation failed: client connection to 127.0.98.62:43351: connect: Connection refused (error 111)
I20260924 04:59:22.545594 7094 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c82275d62aac4bc580771ab80ac00860 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a65aed7f2fdc4e6ea4e8765bbe365a42 in term 2.
I20260924 04:59:22.545662 7795 raft_consensus.cc:2726] T 00000000000000000000000000000000 P c82275d62aac4bc580771ab80ac00860 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 04:59:22.545728 7254 leader_election.cc:302] T 00000000000000000000000000000000 P a65aed7f2fdc4e6ea4e8765bbe365a42 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a65aed7f2fdc4e6ea4e8765bbe365a42, c82275d62aac4bc580771ab80ac00860; no voters: 221d34b534b046078e9003aff8289b32
I20260924 04:59:22.545811 7794 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a65aed7f2fdc4e6ea4e8765bbe365a42 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 04:59:22.545912 7794 raft_consensus.cc:686] T 00000000000000000000000000000000 P a65aed7f2fdc4e6ea4e8765bbe365a42 [term 2 LEADER]: Becoming Leader. State: Replica: a65aed7f2fdc4e6ea4e8765bbe365a42, State: Running, Role: LEADER
I20260924 04:59:22.545972 7794 consensus_queue.cc:237] T 00000000000000000000000000000000 P a65aed7f2fdc4e6ea4e8765bbe365a42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "221d34b534b046078e9003aff8289b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43351 } } peers { permanent_uuid: "c82275d62aac4bc580771ab80ac00860" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40695 } attrs { promote: false } } peers { permanent_uuid: "a65aed7f2fdc4e6ea4e8765bbe365a42" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44097 } attrs { promote: false } }
I20260924 04:59:22.546257 7805 sys_catalog.cc:455] T 00000000000000000000000000000000 P a65aed7f2fdc4e6ea4e8765bbe365a42 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a65aed7f2fdc4e6ea4e8765bbe365a42. Latest consensus state: current_term: 2 leader_uuid: "a65aed7f2fdc4e6ea4e8765bbe365a42" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "221d34b534b046078e9003aff8289b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43351 } } peers { permanent_uuid: "c82275d62aac4bc580771ab80ac00860" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40695 } attrs { promote: false } } peers { permanent_uuid: "a65aed7f2fdc4e6ea4e8765bbe365a42" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44097 } attrs { promote: false } } }
I20260924 04:59:22.546326 7805 sys_catalog.cc:458] T 00000000000000000000000000000000 P a65aed7f2fdc4e6ea4e8765bbe365a42 [sys.catalog]: This master's current role is: LEADER
I20260924 04:59:22.546454 7807 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:59:22.546836 7807 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:59:22.546891 7807 catalog_manager.cc:1259] Loaded cluster ID: c56b85dc80bd450c8c3816861b8b9cb0
I20260924 04:59:22.546924 7807 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:59:22.547096 7807 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:59:22.547163 7807 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a65aed7f2fdc4e6ea4e8765bbe365a42: Loaded TSK: 0
I20260924 04:59:22.547308 7807 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:59:22.587646 7270 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47302:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:59:22.589129 7270 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:59:22.590121 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:22.590004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43351 (local address 127.0.0.1:34916)
0924 04:59:22.590052 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:22.590087 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260924 04:59:22.590163 7094 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c82275d62aac4bc580771ab80ac00860 [term 2 FOLLOWER]: Refusing update from remote peer a65aed7f2fdc4e6ea4e8765bbe365a42: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260924 04:59:22.590483 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:22.590389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43351 (local address 127.0.0.1:34920)
0924 04:59:22.590436 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:22.590466 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260924 04:59:22.590423 7805 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a65aed7f2fdc4e6ea4e8765bbe365a42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c82275d62aac4bc580771ab80ac00860" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40695 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260924 04:59:22.590571 7257 consensus_peers.cc:487] T 00000000000000000000000000000000 P a65aed7f2fdc4e6ea4e8765bbe365a42 -> Peer 221d34b534b046078e9003aff8289b32 (127.0.98.62:43351): Couldn't send request to peer 221d34b534b046078e9003aff8289b32. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43351: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 04:59:22.591010 7795 sys_catalog.cc:455] T 00000000000000000000000000000000 P c82275d62aac4bc580771ab80ac00860 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a65aed7f2fdc4e6ea4e8765bbe365a42. Latest consensus state: current_term: 2 leader_uuid: "a65aed7f2fdc4e6ea4e8765bbe365a42" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "221d34b534b046078e9003aff8289b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43351 } } peers { permanent_uuid: "c82275d62aac4bc580771ab80ac00860" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40695 } attrs { promote: false } } peers { permanent_uuid: "a65aed7f2fdc4e6ea4e8765bbe365a42" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44097 } attrs { promote: false } } }
I20260924 04:59:22.591097 7795 sys_catalog.cc:458] T 00000000000000000000000000000000 P c82275d62aac4bc580771ab80ac00860 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:59:22.591491 7805 sys_catalog.cc:455] T 00000000000000000000000000000000 P a65aed7f2fdc4e6ea4e8765bbe365a42 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "a65aed7f2fdc4e6ea4e8765bbe365a42" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "221d34b534b046078e9003aff8289b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43351 } } peers { permanent_uuid: "c82275d62aac4bc580771ab80ac00860" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40695 } attrs { promote: false } } peers { permanent_uuid: "a65aed7f2fdc4e6ea4e8765bbe365a42" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44097 } attrs { promote: false } } }
I20260924 04:59:22.591583 7805 sys_catalog.cc:458] T 00000000000000000000000000000000 P a65aed7f2fdc4e6ea4e8765bbe365a42 [sys.catalog]: This master's current role is: LEADER
I20260924 04:59:22.591598 7795 sys_catalog.cc:455] T 00000000000000000000000000000000 P c82275d62aac4bc580771ab80ac00860 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "a65aed7f2fdc4e6ea4e8765bbe365a42" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "221d34b534b046078e9003aff8289b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43351 } } peers { permanent_uuid: "c82275d62aac4bc580771ab80ac00860" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40695 } attrs { promote: false } } peers { permanent_uuid: "a65aed7f2fdc4e6ea4e8765bbe365a42" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44097 } attrs { promote: false } } }
I20260924 04:59:22.591681 7795 sys_catalog.cc:458] T 00000000000000000000000000000000 P c82275d62aac4bc580771ab80ac00860 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 04:59:22.592166 7809 mvcc.cc:205] Tried to move back new op lower bound from 7332765542767222784 to 7332765542588813312. Current Snapshot: MvccSnapshot[applied={T|T < 7332765542767222784}]
I20260924 04:59:22.596093 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:22.593132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44049 (local address 127.0.0.1:57232)
0924 04:59:22.593163 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:22.593173 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:22.593211 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:22.593473 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:22.593474 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:22.593475 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:59:22.593524 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:22.593526 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:22.594236 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:22.594237 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:22.595167 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:22.595169 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:22.595887 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:22.595888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:22.595940 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:22.595944 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 04:59:22.595988 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 04:59:22.596031 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18,"mutex_wait_us":2}
I20260924 04:59:22.596179 7796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:22.593189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42241 (local address 127.0.0.1:33634)
0924 04:59:22.593264 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:22.593271 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:22.593303 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:22.593678 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:22.593678 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:22.593679 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:59:22.593727 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:22.593728 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:22.594673 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:22.594673 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:22.595396 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:22.595398 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:22.595979 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:22.595980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:22.596050 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:22.596053 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 04:59:22.596080 (+ 27us) client_negotiation.cc:240] Negotiation successful
0924 04:59:22.596114 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65,"mutex_wait_us":1}
I20260924 04:59:22.596302 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:22.593229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57232 (local address 127.0.98.3:44049)
0924 04:59:22.593337 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:22.593339 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:22.593348 (+ 9us) server_negotiation.cc:407] Connection header received
0924 04:59:22.593373 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:22.593373 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:22.593384 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:22.593469 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:59:22.593606 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:22.594172 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:22.595257 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:22.595827 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:22.595884 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:22.595898 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:22.596179 (+ 281us) server_negotiation.cc:299] Negotiation successful
0924 04:59:22.596224 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80,"thread_start_us":48,"threads_started":1}
I20260924 04:59:22.596467 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:22.593452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33634 (local address 127.0.98.2:42241)
0924 04:59:22.593555 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:22.593557 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:22.593564 (+ 7us) server_negotiation.cc:407] Connection header received
0924 04:59:22.593585 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:22.593586 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:22.593595 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:22.593785 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:59:22.593817 (+ 32us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:22.594582 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:22.595473 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:22.595889 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:22.596339 (+ 450us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:22.596351 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:22.596369 (+ 18us) server_negotiation.cc:299] Negotiation successful
0924 04:59:22.596404 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84,"thread_start_us":58,"threads_started":1}
I20260924 04:59:22.596585 7657 tablet_service.cc:1467] Processing CreateTablet for tablet 81696cdab6024bffbc8e067ee9b778c7 (DEFAULT_TABLE table=test-table [id=c9ae9604d0a54acaa1cd3ddce268ad37]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:59:22.596626 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:22.593195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46631 (local address 127.0.0.1:50142)
0924 04:59:22.593320 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:22.593329 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:22.593372 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:22.593677 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:22.593677 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:22.593679 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:59:22.593727 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:22.593728 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:22.594633 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:22.594634 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:22.595704 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:22.595706 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:22.596424 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:22.596425 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:22.596486 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:22.596490 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 04:59:22.596534 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 04:59:22.596569 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"mutex_wait_us":5}
I20260924 04:59:22.596714 7657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81696cdab6024bffbc8e067ee9b778c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:59:22.596807 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:22.593375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50142 (local address 127.0.98.1:46631)
0924 04:59:22.593544 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:22.593546 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:22.593553 (+ 7us) server_negotiation.cc:407] Connection header received
0924 04:59:22.593577 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:22.593578 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:22.593587 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:22.593786 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:59:22.593820 (+ 34us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:22.594553 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:22.595800 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:22.596362 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:22.596452 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:22.596463 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:22.596702 (+ 239us) server_negotiation.cc:299] Negotiation successful
0924 04:59:22.596746 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":80,"threads_started":1}
I20260924 04:59:22.597872 7817 tablet_bootstrap.cc:492] T 81696cdab6024bffbc8e067ee9b778c7 P b08a35b7530a47f48a8a4cf653551a6a: Bootstrap starting.
I20260924 04:59:22.597990 7521 tablet_service.cc:1467] Processing CreateTablet for tablet 81696cdab6024bffbc8e067ee9b778c7 (DEFAULT_TABLE table=test-table [id=c9ae9604d0a54acaa1cd3ddce268ad37]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:59:22.598098 7521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81696cdab6024bffbc8e067ee9b778c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:59:22.598234 7817 tablet_bootstrap.cc:654] T 81696cdab6024bffbc8e067ee9b778c7 P b08a35b7530a47f48a8a4cf653551a6a: Neither blocks nor log segments found. Creating new log.
I20260924 04:59:22.598507 7385 tablet_service.cc:1467] Processing CreateTablet for tablet 81696cdab6024bffbc8e067ee9b778c7 (DEFAULT_TABLE table=test-table [id=c9ae9604d0a54acaa1cd3ddce268ad37]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:59:22.598620 7385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81696cdab6024bffbc8e067ee9b778c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:59:22.598763 7817 tablet_bootstrap.cc:492] T 81696cdab6024bffbc8e067ee9b778c7 P b08a35b7530a47f48a8a4cf653551a6a: No bootstrap required, opened a new log
I20260924 04:59:22.598805 7817 ts_tablet_manager.cc:1397] T 81696cdab6024bffbc8e067ee9b778c7 P b08a35b7530a47f48a8a4cf653551a6a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:59:22.598917 7817 raft_consensus.cc:348] T 81696cdab6024bffbc8e067ee9b778c7 P b08a35b7530a47f48a8a4cf653551a6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b08a35b7530a47f48a8a4cf653551a6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44049 } } peers { permanent_uuid: "fc45fc89660947038800b5a9922ea04c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42241 } } peers { permanent_uuid: "2b2dd144b90544c5bdbefde04c6672d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46631 } }
I20260924 04:59:22.598973 7817 raft_consensus.cc:374] T 81696cdab6024bffbc8e067ee9b778c7 P b08a35b7530a47f48a8a4cf653551a6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:59:22.598990 7817 raft_consensus.cc:729] T 81696cdab6024bffbc8e067ee9b778c7 P b08a35b7530a47f48a8a4cf653551a6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b08a35b7530a47f48a8a4cf653551a6a, State: Initialized, Role: FOLLOWER
I20260924 04:59:22.599046 7817 consensus_queue.cc:260] T 81696cdab6024bffbc8e067ee9b778c7 P b08a35b7530a47f48a8a4cf653551a6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b08a35b7530a47f48a8a4cf653551a6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44049 } } peers { permanent_uuid: "fc45fc89660947038800b5a9922ea04c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42241 } } peers { permanent_uuid: "2b2dd144b90544c5bdbefde04c6672d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46631 } }
I20260924 04:59:22.599166 7817 ts_tablet_manager.cc:1428] T 81696cdab6024bffbc8e067ee9b778c7 P b08a35b7530a47f48a8a4cf653551a6a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:22.599438 7820 tablet_bootstrap.cc:492] T 81696cdab6024bffbc8e067ee9b778c7 P fc45fc89660947038800b5a9922ea04c: Bootstrap starting.
I20260924 04:59:22.599772 7820 tablet_bootstrap.cc:654] T 81696cdab6024bffbc8e067ee9b778c7 P fc45fc89660947038800b5a9922ea04c: Neither blocks nor log segments found. Creating new log.
I20260924 04:59:22.599779 7821 tablet_bootstrap.cc:492] T 81696cdab6024bffbc8e067ee9b778c7 P 2b2dd144b90544c5bdbefde04c6672d5: Bootstrap starting.
I20260924 04:59:22.600096 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:22.599898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43351 (local address 127.0.98.3:56621)
0924 04:59:22.600028 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:22.600063 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":74,"threads_started":1}
I20260924 04:59:22.600121 7821 tablet_bootstrap.cc:654] T 81696cdab6024bffbc8e067ee9b778c7 P 2b2dd144b90544c5bdbefde04c6672d5: Neither blocks nor log segments found. Creating new log.
I20260924 04:59:22.600446 7820 tablet_bootstrap.cc:492] T 81696cdab6024bffbc8e067ee9b778c7 P fc45fc89660947038800b5a9922ea04c: No bootstrap required, opened a new log
I20260924 04:59:22.600502 7820 ts_tablet_manager.cc:1397] T 81696cdab6024bffbc8e067ee9b778c7 P fc45fc89660947038800b5a9922ea04c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:59:22.600582 7723 heartbeater.cc:503] Master 127.0.98.60:44097 was elected leader, sending a full tablet report...
I20260924 04:59:22.600621 7820 raft_consensus.cc:348] T 81696cdab6024bffbc8e067ee9b778c7 P fc45fc89660947038800b5a9922ea04c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b08a35b7530a47f48a8a4cf653551a6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44049 } } peers { permanent_uuid: "fc45fc89660947038800b5a9922ea04c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42241 } } peers { permanent_uuid: "2b2dd144b90544c5bdbefde04c6672d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46631 } }
I20260924 04:59:22.600683 7820 raft_consensus.cc:374] T 81696cdab6024bffbc8e067ee9b778c7 P fc45fc89660947038800b5a9922ea04c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:59:22.600682 7821 tablet_bootstrap.cc:492] T 81696cdab6024bffbc8e067ee9b778c7 P 2b2dd144b90544c5bdbefde04c6672d5: No bootstrap required, opened a new log
I20260924 04:59:22.600703 7820 raft_consensus.cc:729] T 81696cdab6024bffbc8e067ee9b778c7 P fc45fc89660947038800b5a9922ea04c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc45fc89660947038800b5a9922ea04c, State: Initialized, Role: FOLLOWER
I20260924 04:59:22.600742 7820 consensus_queue.cc:260] T 81696cdab6024bffbc8e067ee9b778c7 P fc45fc89660947038800b5a9922ea04c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b08a35b7530a47f48a8a4cf653551a6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44049 } } peers { permanent_uuid: "fc45fc89660947038800b5a9922ea04c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42241 } } peers { permanent_uuid: "2b2dd144b90544c5bdbefde04c6672d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46631 } }
I20260924 04:59:22.600872 7820 ts_tablet_manager.cc:1428] T 81696cdab6024bffbc8e067ee9b778c7 P fc45fc89660947038800b5a9922ea04c: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 04:59:22.600724 7821 ts_tablet_manager.cc:1397] T 81696cdab6024bffbc8e067ee9b778c7 P 2b2dd144b90544c5bdbefde04c6672d5: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:59:22.601254 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:22.601032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43351 (local address 127.0.98.2:55471)
0924 04:59:22.601180 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:22.601216 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":79,"threads_started":1}
I20260924 04:59:22.601255 7821 raft_consensus.cc:348] T 81696cdab6024bffbc8e067ee9b778c7 P 2b2dd144b90544c5bdbefde04c6672d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b08a35b7530a47f48a8a4cf653551a6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44049 } } peers { permanent_uuid: "fc45fc89660947038800b5a9922ea04c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42241 } } peers { permanent_uuid: "2b2dd144b90544c5bdbefde04c6672d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46631 } }
I20260924 04:59:22.601372 7821 raft_consensus.cc:374] T 81696cdab6024bffbc8e067ee9b778c7 P 2b2dd144b90544c5bdbefde04c6672d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:59:22.601400 7821 raft_consensus.cc:729] T 81696cdab6024bffbc8e067ee9b778c7 P 2b2dd144b90544c5bdbefde04c6672d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b2dd144b90544c5bdbefde04c6672d5, State: Initialized, Role: FOLLOWER
I20260924 04:59:22.601475 7821 consensus_queue.cc:260] T 81696cdab6024bffbc8e067ee9b778c7 P 2b2dd144b90544c5bdbefde04c6672d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b08a35b7530a47f48a8a4cf653551a6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44049 } } peers { permanent_uuid: "fc45fc89660947038800b5a9922ea04c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42241 } } peers { permanent_uuid: "2b2dd144b90544c5bdbefde04c6672d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46631 } }
I20260924 04:59:22.601629 7821 ts_tablet_manager.cc:1428] T 81696cdab6024bffbc8e067ee9b778c7 P 2b2dd144b90544c5bdbefde04c6672d5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:22.602015 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:22.601811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43351 (local address 127.0.98.1:51955)
0924 04:59:22.601956 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:22.601988 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":74,"threads_started":1}
I20260924 04:59:22.602084 7451 heartbeater.cc:503] Master 127.0.98.60:44097 was elected leader, sending a full tablet report...
W20260924 04:59:22.619127 7454 tablet.cc:2369] T 81696cdab6024bffbc8e067ee9b778c7 P 2b2dd144b90544c5bdbefde04c6672d5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 04:59:22.634810 7726 tablet.cc:2369] T 81696cdab6024bffbc8e067ee9b778c7 P b08a35b7530a47f48a8a4cf653551a6a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:59:22.668164 7825 raft_consensus.cc:482] T 81696cdab6024bffbc8e067ee9b778c7 P fc45fc89660947038800b5a9922ea04c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:59:22.668251 7825 raft_consensus.cc:504] T 81696cdab6024bffbc8e067ee9b778c7 P fc45fc89660947038800b5a9922ea04c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b08a35b7530a47f48a8a4cf653551a6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44049 } } peers { permanent_uuid: "fc45fc89660947038800b5a9922ea04c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42241 } } peers { permanent_uuid: "2b2dd144b90544c5bdbefde04c6672d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46631 } }
I20260924 04:59:22.668602 7825 leader_election.cc:288] T 81696cdab6024bffbc8e067ee9b778c7 P fc45fc89660947038800b5a9922ea04c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b08a35b7530a47f48a8a4cf653551a6a (127.0.98.3:44049), 2b2dd144b90544c5bdbefde04c6672d5 (127.0.98.1:46631)
I20260924 04:59:22.671211 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:22.668825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42633 (local address 127.0.98.1:46631)
0924 04:59:22.668879 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:22.668881 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:22.668891 (+ 10us) server_negotiation.cc:407] Connection header received
0924 04:59:22.668943 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:22.668944 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:22.668965 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:22.669005 (+ 40us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:59:22.669134 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:22.669660 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:22.670583 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:22.670917 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:22.670984 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:22.670996 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:22.671113 (+ 117us) server_negotiation.cc:299] Negotiation successful
0924 04:59:22.671146 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 04:59:22.671209 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:22.668748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46631 (local address 127.0.98.2:42633)
0924 04:59:22.668810 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:22.668822 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:22.668883 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:22.669017 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:22.669018 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:22.669020 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:59:22.669077 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:22.669079 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:22.669730 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:22.669731 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:22.670497 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:22.670499 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:22.670989 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:22.670990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:22.671052 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:22.671056 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 04:59:22.671099 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 04:59:22.671134 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260924 04:59:22.671296 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:22.668798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36551 (local address 127.0.98.3:44049)
0924 04:59:22.668891 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:22.668894 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:22.669031 (+ 137us) server_negotiation.cc:407] Connection header received
0924 04:59:22.669079 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:22.669080 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:22.669091 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:22.669136 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:59:22.669259 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:22.669763 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:22.670647 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:22.671003 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:22.671060 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:22.671071 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:22.671203 (+ 132us) server_negotiation.cc:299] Negotiation successful
0924 04:59:22.671234 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 04:59:22.671363 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:22.668749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44049 (local address 127.0.98.2:36551)
0924 04:59:22.668962 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:22.668972 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:22.669024 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:22.669142 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:22.669143 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:22.669144 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:59:22.669203 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:22.669204 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:22.669831 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:22.669832 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:22.670575 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:22.670577 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:22.671092 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:22.671093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:22.671147 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:22.671150 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 04:59:22.671193 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 04:59:22.671230 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"mutex_wait_us":53,"thread_start_us":113,"threads_started":1}
I20260924 04:59:22.671628 7405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81696cdab6024bffbc8e067ee9b778c7" candidate_uuid: "fc45fc89660947038800b5a9922ea04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b2dd144b90544c5bdbefde04c6672d5" is_pre_election: true
I20260924 04:59:22.671725 7405 raft_consensus.cc:2454] T 81696cdab6024bffbc8e067ee9b778c7 P 2b2dd144b90544c5bdbefde04c6672d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc45fc89660947038800b5a9922ea04c in term 0.
I20260924 04:59:22.671717 7677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81696cdab6024bffbc8e067ee9b778c7" candidate_uuid: "fc45fc89660947038800b5a9922ea04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b08a35b7530a47f48a8a4cf653551a6a" is_pre_election: true
I20260924 04:59:22.671844 7677 raft_consensus.cc:2454] T 81696cdab6024bffbc8e067ee9b778c7 P b08a35b7530a47f48a8a4cf653551a6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc45fc89660947038800b5a9922ea04c in term 0.
I20260924 04:59:22.671890 7477 leader_election.cc:302] T 81696cdab6024bffbc8e067ee9b778c7 P fc45fc89660947038800b5a9922ea04c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b2dd144b90544c5bdbefde04c6672d5, fc45fc89660947038800b5a9922ea04c; no voters:
I20260924 04:59:22.672050 7825 raft_consensus.cc:2781] T 81696cdab6024bffbc8e067ee9b778c7 P fc45fc89660947038800b5a9922ea04c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:59:22.672092 7825 raft_consensus.cc:482] T 81696cdab6024bffbc8e067ee9b778c7 P fc45fc89660947038800b5a9922ea04c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:59:22.672116 7825 raft_consensus.cc:3037] T 81696cdab6024bffbc8e067ee9b778c7 P fc45fc89660947038800b5a9922ea04c [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:59:22.672742 7825 raft_consensus.cc:504] T 81696cdab6024bffbc8e067ee9b778c7 P fc45fc89660947038800b5a9922ea04c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b08a35b7530a47f48a8a4cf653551a6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44049 } } peers { permanent_uuid: "fc45fc89660947038800b5a9922ea04c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42241 } } peers { permanent_uuid: "2b2dd144b90544c5bdbefde04c6672d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46631 } }
I20260924 04:59:22.672860 7825 leader_election.cc:288] T 81696cdab6024bffbc8e067ee9b778c7 P fc45fc89660947038800b5a9922ea04c [CANDIDATE]: Term 1 election: Requested vote from peers b08a35b7530a47f48a8a4cf653551a6a (127.0.98.3:44049), 2b2dd144b90544c5bdbefde04c6672d5 (127.0.98.1:46631)
I20260924 04:59:22.673007 7405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81696cdab6024bffbc8e067ee9b778c7" candidate_uuid: "fc45fc89660947038800b5a9922ea04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b2dd144b90544c5bdbefde04c6672d5"
I20260924 04:59:22.673029 7677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81696cdab6024bffbc8e067ee9b778c7" candidate_uuid: "fc45fc89660947038800b5a9922ea04c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b08a35b7530a47f48a8a4cf653551a6a"
I20260924 04:59:22.673087 7677 raft_consensus.cc:3037] T 81696cdab6024bffbc8e067ee9b778c7 P b08a35b7530a47f48a8a4cf653551a6a [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:59:22.673087 7405 raft_consensus.cc:3037] T 81696cdab6024bffbc8e067ee9b778c7 P 2b2dd144b90544c5bdbefde04c6672d5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:59:22.673830 7405 raft_consensus.cc:2454] T 81696cdab6024bffbc8e067ee9b778c7 P 2b2dd144b90544c5bdbefde04c6672d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc45fc89660947038800b5a9922ea04c in term 1.
I20260924 04:59:22.673899 7677 raft_consensus.cc:2454] T 81696cdab6024bffbc8e067ee9b778c7 P b08a35b7530a47f48a8a4cf653551a6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc45fc89660947038800b5a9922ea04c in term 1.
I20260924 04:59:22.674054 7474 leader_election.cc:302] T 81696cdab6024bffbc8e067ee9b778c7 P fc45fc89660947038800b5a9922ea04c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b08a35b7530a47f48a8a4cf653551a6a, fc45fc89660947038800b5a9922ea04c; no voters:
I20260924 04:59:22.674153 7825 raft_consensus.cc:2781] T 81696cdab6024bffbc8e067ee9b778c7 P fc45fc89660947038800b5a9922ea04c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:59:22.674296 7825 raft_consensus.cc:686] T 81696cdab6024bffbc8e067ee9b778c7 P fc45fc89660947038800b5a9922ea04c [term 1 LEADER]: Becoming Leader. State: Replica: fc45fc89660947038800b5a9922ea04c, State: Running, Role: LEADER
I20260924 04:59:22.674379 7587 heartbeater.cc:503] Master 127.0.98.60:44097 was elected leader, sending a full tablet report...
I20260924 04:59:22.674366 7825 consensus_queue.cc:237] T 81696cdab6024bffbc8e067ee9b778c7 P fc45fc89660947038800b5a9922ea04c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b08a35b7530a47f48a8a4cf653551a6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44049 } } peers { permanent_uuid: "fc45fc89660947038800b5a9922ea04c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42241 } } peers { permanent_uuid: "2b2dd144b90544c5bdbefde04c6672d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46631 } }
I20260924 04:59:22.674625 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:22.674499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43351 (local address 127.0.98.2:40291)
0924 04:59:22.674561 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:22.674597 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260924 04:59:22.675122 7270 catalog_manager.cc:5495] T 81696cdab6024bffbc8e067ee9b778c7 P fc45fc89660947038800b5a9922ea04c reported cstate change: term changed from 0 to 1, leader changed from <none> to fc45fc89660947038800b5a9922ea04c (127.0.98.2). New cstate: current_term: 1 leader_uuid: "fc45fc89660947038800b5a9922ea04c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b08a35b7530a47f48a8a4cf653551a6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fc45fc89660947038800b5a9922ea04c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42241 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2b2dd144b90544c5bdbefde04c6672d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46631 } health_report { overall_health: UNKNOWN } } }
I20260924 04:59:22.689309 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:22.687187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33642 (local address 127.0.98.2:42241)
0924 04:59:22.687223 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:22.687225 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:22.687235 (+ 10us) server_negotiation.cc:407] Connection header received
0924 04:59:22.687313 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:22.687313 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:22.687333 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:22.687391 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:59:22.687512 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:22.687974 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:22.688665 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:22.688847 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:22.688898 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:22.689027 (+ 129us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:59:22.689121 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:59:22.689180 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:22.689216 (+ 36us) server_negotiation.cc:299] Negotiation successful
0924 04:59:22.689236 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 04:59:22.690201 7405 raft_consensus.cc:1261] T 81696cdab6024bffbc8e067ee9b778c7 P 2b2dd144b90544c5bdbefde04c6672d5 [term 1 FOLLOWER]: Refusing update from remote peer fc45fc89660947038800b5a9922ea04c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:59:22.690282 7677 raft_consensus.cc:1261] T 81696cdab6024bffbc8e067ee9b778c7 P b08a35b7530a47f48a8a4cf653551a6a [term 1 FOLLOWER]: Refusing update from remote peer fc45fc89660947038800b5a9922ea04c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:59:22.690414 7825 consensus_queue.cc:1035] T 81696cdab6024bffbc8e067ee9b778c7 P fc45fc89660947038800b5a9922ea04c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b2dd144b90544c5bdbefde04c6672d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:59:22.690532 7830 consensus_queue.cc:1035] T 81696cdab6024bffbc8e067ee9b778c7 P fc45fc89660947038800b5a9922ea04c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b08a35b7530a47f48a8a4cf653551a6a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:59:22.691718 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:22.691470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43351 (local address 127.0.98.1:43137)
0924 04:59:22.691656 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:22.691690 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169}
I20260924 04:59:22.691802 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:22.691695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43351 (local address 127.0.98.3:56341)
0924 04:59:22.691746 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:22.691776 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260924 04:59:22.692476 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:22.692355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43351 (local address 127.0.98.2:36521)
0924 04:59:22.692410 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:22.692434 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260924 04:59:22.693085 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:22.692787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43351 (local address 127.0.98.3:43541)
0924 04:59:22.693028 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:22.693058 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224}
I20260924 04:59:22.693105 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:22.692999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43351 (local address 127.0.98.2:41839)
0924 04:59:22.693056 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:22.693079 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260924 04:59:22.693498 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:22.693394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43351 (local address 127.0.98.1:55493)
0924 04:59:22.693448 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:22.693472 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260924 04:59:22.695055 7270 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47302:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:59:22.695129 7270 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47302:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:59:22.696169 7270 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a65aed7f2fdc4e6ea4e8765bbe365a42: Sending DeleteTablet for 3 replicas of tablet 81696cdab6024bffbc8e067ee9b778c7
I20260924 04:59:22.696467 7521 tablet_service.cc:1514] Processing DeleteTablet for tablet 81696cdab6024bffbc8e067ee9b778c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:59:22 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:33634
I20260924 04:59:22.696668 7385 tablet_service.cc:1514] Processing DeleteTablet for tablet 81696cdab6024bffbc8e067ee9b778c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:59:22 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:50142
I20260924 04:59:22.696732 7657 tablet_service.cc:1514] Processing DeleteTablet for tablet 81696cdab6024bffbc8e067ee9b778c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:59:22 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:57232
I20260924 04:59:22.696889 7844 tablet_replica.cc:330] T 81696cdab6024bffbc8e067ee9b778c7 P b08a35b7530a47f48a8a4cf653551a6a: stopping tablet replica
I20260924 04:59:22.696982 7844 raft_consensus.cc:2229] T 81696cdab6024bffbc8e067ee9b778c7 P b08a35b7530a47f48a8a4cf653551a6a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:59:22.697028 7843 tablet_replica.cc:330] T 81696cdab6024bffbc8e067ee9b778c7 P 2b2dd144b90544c5bdbefde04c6672d5: stopping tablet replica
I20260924 04:59:22.697090 7843 raft_consensus.cc:2229] T 81696cdab6024bffbc8e067ee9b778c7 P 2b2dd144b90544c5bdbefde04c6672d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:59:22.697136 7843 raft_consensus.cc:2258] T 81696cdab6024bffbc8e067ee9b778c7 P 2b2dd144b90544c5bdbefde04c6672d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:59:22.697146 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 7322
I20260924 04:59:22.697156 7844 raft_consensus.cc:2258] T 81696cdab6024bffbc8e067ee9b778c7 P b08a35b7530a47f48a8a4cf653551a6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:59:22.697228 7842 tablet_replica.cc:330] T 81696cdab6024bffbc8e067ee9b778c7 P fc45fc89660947038800b5a9922ea04c: stopping tablet replica
I20260924 04:59:22.697273 7842 raft_consensus.cc:2229] T 81696cdab6024bffbc8e067ee9b778c7 P fc45fc89660947038800b5a9922ea04c [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:59:22.697355 7842 raft_consensus.cc:2258] T 81696cdab6024bffbc8e067ee9b778c7 P fc45fc89660947038800b5a9922ea04c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:59:22.697943 7844 ts_tablet_manager.cc:1905] T 81696cdab6024bffbc8e067ee9b778c7 P b08a35b7530a47f48a8a4cf653551a6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:59:22.697990 7842 ts_tablet_manager.cc:1905] T 81696cdab6024bffbc8e067ee9b778c7 P fc45fc89660947038800b5a9922ea04c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:59:22.699700 7842 ts_tablet_manager.cc:1918] T 81696cdab6024bffbc8e067ee9b778c7 P fc45fc89660947038800b5a9922ea04c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:59:22.699723 7844 ts_tablet_manager.cc:1918] T 81696cdab6024bffbc8e067ee9b778c7 P b08a35b7530a47f48a8a4cf653551a6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:59:22.699757 7842 log.cc:1196] T 81696cdab6024bffbc8e067ee9b778c7 P fc45fc89660947038800b5a9922ea04c: Deleting WAL directory at /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/ts-1/wal/wals/81696cdab6024bffbc8e067ee9b778c7
I20260924 04:59:22.699771 7844 log.cc:1196] T 81696cdab6024bffbc8e067ee9b778c7 P b08a35b7530a47f48a8a4cf653551a6a: Deleting WAL directory at /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790225929848825-392-0/minicluster-data/ts-2/wal/wals/81696cdab6024bffbc8e067ee9b778c7
I20260924 04:59:22.699996 7842 ts_tablet_manager.cc:1939] T 81696cdab6024bffbc8e067ee9b778c7 P fc45fc89660947038800b5a9922ea04c: Deleting consensus metadata
I20260924 04:59:22.700006 7844 ts_tablet_manager.cc:1939] T 81696cdab6024bffbc8e067ee9b778c7 P b08a35b7530a47f48a8a4cf653551a6a: Deleting consensus metadata
I20260924 04:59:22.700261 7256 catalog_manager.cc:4841] TS fc45fc89660947038800b5a9922ea04c (127.0.98.2:42241): tablet 81696cdab6024bffbc8e067ee9b778c7 (table test-table [id=c9ae9604d0a54acaa1cd3ddce268ad37]) successfully deleted
I20260924 04:59:22.700271 7254 catalog_manager.cc:4841] TS b08a35b7530a47f48a8a4cf653551a6a (127.0.98.3:44049): tablet 81696cdab6024bffbc8e067ee9b778c7 (table test-table [id=c9ae9604d0a54acaa1cd3ddce268ad37]) successfully deleted
I20260924 04:59:22.703370 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:22.703268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46631 (local address 127.0.0.1:50158)
0924 04:59:22.703315 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:22.703340 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
W20260924 04:59:22.703491 7257 catalog_manager.cc:4584] TS 2b2dd144b90544c5bdbefde04c6672d5 (127.0.98.1:46631): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 81696cdab6024bffbc8e067ee9b778c7: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46631: connect: Connection refused (error 111)
I20260924 04:59:22.703586 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 7458
I20260924 04:59:22.708055 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 7594
I20260924 04:59:22.712653 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 6951
I20260924 04:59:22.716383 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 7138
2026-09-24T04:59:22Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (4511 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225929848825-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:59:22 dist-test-slave-70st krb5kdc[7847](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:59:22 dist-test-slave-70st krb5kdc[7847](info): set up 2 sockets
Sep 24 04:59:22 dist-test-slave-70st krb5kdc[7847](info): commencing operation
krb5kdc: starting...
W20260924 04:59:23.665716 392 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225929848825-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225929848825-392-0/krb5kdc/test-user.keytab.
Sep 24 04:59:23 dist-test-slave-70st krb5kdc[7847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225963, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:59:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:59:23Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225929848825-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225929848825-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260924 04:59:23.776773 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225929848825-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225929848825-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225929848825-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44523
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225929848825-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44523
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:59:23.854485 7860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:59:23.854681 7860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:59:23.854701 7860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:59:23.855871 7860 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:59:23.855912 7860 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 04:59:23.855924 7860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:59:23.855935 7860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:59:23.855947 7860 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:59:23.855957 7860 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:59:23.857399 7860 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225929848825-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225929848825-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44523
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225929848825-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44523
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7860
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225929848825-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 04:59:23.857640 7860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:59:23.857841 7860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:59:23.862238 7866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:59:23.862254 7868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:59:23.862237 7865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:59:23.862497 7860 server_base.cc:1034] running on GCE node
I20260924 04:59:23.862733 7860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:59:23.862999 7860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:59:23.864140 7860 hybrid_clock.cc:648] HybridClock initialized: now 1790225963864127 us; error 24 us; skew 500 ppm
Sep 24 04:59:23 dist-test-slave-70st krb5kdc[7847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225963, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:59:23.866412 7860 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260924 04:59:23.866806 7860 webserver.cc:460] Webserver started at http://127.0.98.62:34021/ using document root <none> and password file <none>
I20260924 04:59:23.867020 7860 fs_manager.cc:359] Metadata directory not provided
I20260924 04:59:23.867067 7860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:59:23.867182 7860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:59:23.868155 7860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225929848825-392-0/minicluster-data/master-0/data/instance:
uuid: "8d8dc2f2261142e59c7c7ca2b072e3fd"
format_stamp: "Formatted at 2026-09-24 04:59:23 on dist-test-slave-70st"
server_key: "2bd7fba39046e96ef2095cb5d62fa8dd"
server_key_iv: "5deae57bf4ad7fe04f58589081f958bc"
server_key_version: "encryptionkey@0"
I20260924 04:59:23.868458 7860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225929848825-392-0/minicluster-data/master-0/wal/instance:
uuid: "8d8dc2f2261142e59c7c7ca2b072e3fd"
format_stamp: "Formatted at 2026-09-24 04:59:23 on dist-test-slave-70st"
server_key: "2bd7fba39046e96ef2095cb5d62fa8dd"
server_key_iv: "5deae57bf4ad7fe04f58589081f958bc"
server_key_version: "encryptionkey@0"
I20260924 04:59:23.869637 7860 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 04:59:23.870271 7875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:23.870405 7860 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:23.870494 7860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225929848825-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225929848825-392-0/minicluster-data/master-0/wal
uuid: "8d8dc2f2261142e59c7c7ca2b072e3fd"
format_stamp: "Formatted at 2026-09-24 04:59:23 on dist-test-slave-70st"
server_key: "2bd7fba39046e96ef2095cb5d62fa8dd"
server_key_iv: "5deae57bf4ad7fe04f58589081f958bc"
server_key_version: "encryptionkey@0"
I20260924 04:59:23.870563 7860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225929848825-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225929848825-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225929848825-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:59:23.880235 7860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:59:23.881898 7860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:59:23.882000 7860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:59:23.886102 7860 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44523
I20260924 04:59:23.886161 7937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44523 every 8 connection(s)
I20260924 04:59:23.886435 7860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:59:23.887077 7938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:59:23.889173 7938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d8dc2f2261142e59c7c7ca2b072e3fd: Bootstrap starting.
I20260924 04:59:23.889735 7938 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d8dc2f2261142e59c7c7ca2b072e3fd: Neither blocks nor log segments found. Creating new log.
I20260924 04:59:23.889976 7938 log.cc:824] T 00000000000000000000000000000000 P 8d8dc2f2261142e59c7c7ca2b072e3fd: Log is configured to *not* fsync() on all Append() calls
I20260924 04:59:23.890604 7938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d8dc2f2261142e59c7c7ca2b072e3fd: No bootstrap required, opened a new log
I20260924 04:59:23.890950 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 7860
I20260924 04:59:23.891053 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225929848825-392-0/minicluster-data/master-0/wal/instance
I20260924 04:59:23.891227 392 external_mini_cluster.cc:1428] Setting key 01fdd189ba6cc344d823769ffc0582f7
I20260924 04:59:23.892922 7938 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d8dc2f2261142e59c7c7ca2b072e3fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d8dc2f2261142e59c7c7ca2b072e3fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44523 } }
I20260924 04:59:23.893049 7938 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d8dc2f2261142e59c7c7ca2b072e3fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:59:23.893070 7938 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d8dc2f2261142e59c7c7ca2b072e3fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d8dc2f2261142e59c7c7ca2b072e3fd, State: Initialized, Role: FOLLOWER
I20260924 04:59:23.893203 7938 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d8dc2f2261142e59c7c7ca2b072e3fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d8dc2f2261142e59c7c7ca2b072e3fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44523 } }
I20260924 04:59:23.893282 7938 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d8dc2f2261142e59c7c7ca2b072e3fd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:59:23.893311 7938 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d8dc2f2261142e59c7c7ca2b072e3fd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:59:23.893356 7938 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d8dc2f2261142e59c7c7ca2b072e3fd [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:59:23.893985 7938 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d8dc2f2261142e59c7c7ca2b072e3fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d8dc2f2261142e59c7c7ca2b072e3fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44523 } }
I20260924 04:59:23.894073 7938 leader_election.cc:302] T 00000000000000000000000000000000 P 8d8dc2f2261142e59c7c7ca2b072e3fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d8dc2f2261142e59c7c7ca2b072e3fd; no voters:
I20260924 04:59:23.894204 7938 leader_election.cc:288] T 00000000000000000000000000000000 P 8d8dc2f2261142e59c7c7ca2b072e3fd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:59:23.894268 7943 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d8dc2f2261142e59c7c7ca2b072e3fd [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:59:23.894436 7943 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d8dc2f2261142e59c7c7ca2b072e3fd [term 1 LEADER]: Becoming Leader. State: Replica: 8d8dc2f2261142e59c7c7ca2b072e3fd, State: Running, Role: LEADER
I20260924 04:59:23.894445 7938 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d8dc2f2261142e59c7c7ca2b072e3fd [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:59:23.894552 7943 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d8dc2f2261142e59c7c7ca2b072e3fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d8dc2f2261142e59c7c7ca2b072e3fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44523 } }
I20260924 04:59:23.894943 7945 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d8dc2f2261142e59c7c7ca2b072e3fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8d8dc2f2261142e59c7c7ca2b072e3fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d8dc2f2261142e59c7c7ca2b072e3fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44523 } } }
I20260924 04:59:23.894956 7944 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d8dc2f2261142e59c7c7ca2b072e3fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d8dc2f2261142e59c7c7ca2b072e3fd. Latest consensus state: current_term: 1 leader_uuid: "8d8dc2f2261142e59c7c7ca2b072e3fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d8dc2f2261142e59c7c7ca2b072e3fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44523 } } }
I20260924 04:59:23.895047 7945 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d8dc2f2261142e59c7c7ca2b072e3fd [sys.catalog]: This master's current role is: LEADER
I20260924 04:59:23.895059 7944 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d8dc2f2261142e59c7c7ca2b072e3fd [sys.catalog]: This master's current role is: LEADER
I20260924 04:59:23.895226 7950 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:59:23.895735 7950 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:59:23.897395 7950 catalog_manager.cc:1347] Generated new cluster ID: 98b510a13a174140bea8cabcaa1bdbce
I20260924 04:59:23.897465 7950 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 04:59:23 dist-test-slave-70st krb5kdc[7847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225963, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 04:59:23.903434 7942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:23.891747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49160 (local address 127.0.98.62:44523)
0924 04:59:23.891915 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:23.891924 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:23.891953 (+ 29us) server_negotiation.cc:407] Connection header received
0924 04:59:23.892911 (+ 958us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:23.892919 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:23.892976 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:23.893130 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:23.893904 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:23.895282 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:23.896304 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:23.896889 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:23.897001 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:23.899899 (+ 2898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:23.899903 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:23.899904 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:59:23.899916 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:23.901885 (+ 1969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:23.902269 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:23.902271 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:23.902272 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:23.902393 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:23.902582 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:23.902582 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:23.902582 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:23.902866 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:23.903011 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:23.903112 (+ 101us) server_negotiation.cc:299] Negotiation successful
0924 04:59:23.903235 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":79,"threads_started":1}
I20260924 04:59:23.907104 7950 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:59:23.907647 7950 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:59:23.924793 7950 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d8dc2f2261142e59c7c7ca2b072e3fd: Generated new TSK 0
I20260924 04:59:23.924979 7950 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225929848825-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225929848825-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260924 04:59:23.955930 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225929848825-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225929848825-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225929848825-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225929848825-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44523
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225929848825-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:59:24.034889 7964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:59:24.035120 7964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:59:24.035142 7964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:59:24.036293 7964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:59:24.036336 7964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:59:24.036356 7964 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 04:59:24.036372 7964 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:59:24.037974 7964 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225929848825-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225929848825-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225929848825-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225929848825-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:44523
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7964
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225929848825-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 04:59:24.038198 7964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:59:24.038424 7964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:59:24.043130 7970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:59:24.043226 7964 server_base.cc:1034] running on GCE node
W20260924 04:59:24.043136 7969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:59:24.043143 7972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:59:24.043573 7964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:59:24.043793 7964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:59:24.044973 7964 hybrid_clock.cc:648] HybridClock initialized: now 1790225964044959 us; error 32 us; skew 500 ppm
Sep 24 04:59:24 dist-test-slave-70st krb5kdc[7847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225964, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:59:24.047387 7964 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260924 04:59:24.047820 7964 webserver.cc:460] Webserver started at http://127.0.98.1:42787/ using document root <none> and password file <none>
I20260924 04:59:24.048023 7964 fs_manager.cc:359] Metadata directory not provided
I20260924 04:59:24.048071 7964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:59:24.048187 7964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:59:24.049196 7964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225929848825-392-0/minicluster-data/ts-0/data/instance:
uuid: "00a93275756742d3b9fa9c06d2bd8da4"
format_stamp: "Formatted at 2026-09-24 04:59:24 on dist-test-slave-70st"
server_key: "0a3eea50aaa1d0972df644a5bc07abba"
server_key_iv: "cbab0ebf4c9ccbb32916e9e12c920ff1"
server_key_version: "encryptionkey@0"
I20260924 04:59:24.049551 7964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225929848825-392-0/minicluster-data/ts-0/wal/instance:
uuid: "00a93275756742d3b9fa9c06d2bd8da4"
format_stamp: "Formatted at 2026-09-24 04:59:24 on dist-test-slave-70st"
server_key: "0a3eea50aaa1d0972df644a5bc07abba"
server_key_iv: "cbab0ebf4c9ccbb32916e9e12c920ff1"
server_key_version: "encryptionkey@0"
I20260924 04:59:24.050788 7964 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 04:59:24.051509 7979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:24.051685 7964 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 04:59:24.051772 7964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225929848825-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225929848825-392-0/minicluster-data/ts-0/wal
uuid: "00a93275756742d3b9fa9c06d2bd8da4"
format_stamp: "Formatted at 2026-09-24 04:59:24 on dist-test-slave-70st"
server_key: "0a3eea50aaa1d0972df644a5bc07abba"
server_key_iv: "cbab0ebf4c9ccbb32916e9e12c920ff1"
server_key_version: "encryptionkey@0"
I20260924 04:59:24.051834 7964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225929848825-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225929848825-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225929848825-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:59:24.074821 7964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:59:24.077081 7964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:59:24.077241 7964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:59:24.077854 7964 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:59:24.077905 7964 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:24.077939 7964 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:59:24.077970 7964 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 04:59:24 dist-test-slave-70st krb5kdc[7847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225964, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 04:59:24.085026 7964 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34321
I20260924 04:59:24.085093 8094 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34321 every 8 connection(s)
I20260924 04:59:24.085489 7964 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225929848825-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:59:24.087725 7942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:24.078304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40463 (local address 127.0.98.62:44523)
0924 04:59:24.078372 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:24.078374 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:24.078594 (+ 220us) server_negotiation.cc:407] Connection header received
0924 04:59:24.079081 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:24.079083 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:24.079102 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:24.079184 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:24.079833 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:24.080575 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:24.081891 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:24.082273 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:24.082353 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:24.084936 (+ 2583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:24.084938 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:24.084938 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:59:24.084947 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:24.086419 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:24.086823 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:24.086824 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:24.086824 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:24.086876 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:24.087097 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:24.087098 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:24.087098 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:24.087216 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:24.087274 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:24.087575 (+ 301us) server_negotiation.cc:299] Negotiation successful
0924 04:59:24.087659 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 04:59:24.087806 7988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:24.078297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44523 (local address 127.0.98.1:40463)
0924 04:59:24.078465 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:24.078506 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:24.078993 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:24.079262 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:24.079266 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:24.079535 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:24.079753 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:24.079758 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:24.080676 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:24.080677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:24.081819 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:24.081822 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:24.082377 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:24.082378 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:24.082499 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:24.083038 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:24.083047 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:24.084847 (+ 1800us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:59:24.086508 (+ 1661us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:24.086515 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:24.086516 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:24.086738 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:24.086933 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:24.086933 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:24.086934 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:24.087045 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:24.087299 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:59:24.087304 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:59:24.087432 (+ 128us) client_negotiation.cc:769] Sending connection context
0924 04:59:24.087520 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 04:59:24.087640 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":88,"threads_started":1}
I20260924 04:59:24.088238 8095 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44523
I20260924 04:59:24.088321 8095 heartbeater.cc:464] Registering TS with master...
I20260924 04:59:24.088483 8095 heartbeater.cc:511] Master 127.0.98.62:44523 requested a full tablet report, sending...
I20260924 04:59:24.089056 7891 ts_manager.cc:194] Registered new tserver with Master: 00a93275756742d3b9fa9c06d2bd8da4 (127.0.98.1:34321)
I20260924 04:59:24.089753 7891 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:40463
I20260924 04:59:24.090559 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 7964
I20260924 04:59:24.090677 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225929848825-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:59:24.090893 392 external_mini_cluster.cc:1428] Setting key 2014c07a808bfabd07dc6e8f962d8190
I20260924 04:59:24.094867 7942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:24.089218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48735 (local address 127.0.98.62:44523)
0924 04:59:24.089258 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:24.089260 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:24.089342 (+ 82us) server_negotiation.cc:407] Connection header received
0924 04:59:24.089405 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:24.089405 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:24.089434 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:24.089470 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:24.089753 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:24.090382 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:24.091362 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:24.091638 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:24.091697 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:24.092590 (+ 893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:24.092591 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:24.092592 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:59:24.092598 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:24.093858 (+ 1260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:24.094144 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:24.094145 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:24.094145 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:24.094235 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:24.094398 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:24.094399 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:24.094399 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:24.094502 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:24.094553 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:24.094696 (+ 143us) server_negotiation.cc:299] Negotiation successful
0924 04:59:24.094767 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 04:59:24.094875 8101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:24.089175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44523 (local address 127.0.98.1:48735)
0924 04:59:24.089296 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:24.089304 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:24.089354 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:24.089484 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:24.089485 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:24.089634 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:24.089690 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:24.089692 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:24.090479 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:24.090480 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:24.091266 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:24.091268 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:24.091718 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:24.091719 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:24.091790 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:24.092182 (+ 392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:24.092187 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:24.092509 (+ 322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:59:24.093930 (+ 1421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:24.093931 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:24.093931 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:24.094086 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:24.094287 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:24.094288 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:24.094288 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:24.094350 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:24.094555 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:59:24.094555 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:59:24.094610 (+ 55us) client_negotiation.cc:769] Sending connection context
0924 04:59:24.094667 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 04:59:24.094767 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":61,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225929848825-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225929848825-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260924 04:59:24.117290 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225929848825-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225929848825-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225929848825-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225929848825-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44523
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225929848825-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:59:24.198537 8104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:59:24.198750 8104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:59:24.198772 8104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:59:24.199913 8104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:59:24.199952 8104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:59:24.199971 8104 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 04:59:24.199990 8104 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:59:24.201490 8104 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225929848825-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225929848825-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225929848825-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225929848825-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:44523
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8104
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225929848825-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 04:59:24.201700 8104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:59:24.201910 8104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:59:24.206566 8110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:59:24.206575 8112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:59:24.206575 8109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:59:24.206864 8104 server_base.cc:1034] running on GCE node
I20260924 04:59:24.207031 8104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:59:24.207231 8104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:59:24.208381 8104 hybrid_clock.cc:648] HybridClock initialized: now 1790225964208359 us; error 34 us; skew 500 ppm
Sep 24 04:59:24 dist-test-slave-70st krb5kdc[7847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225964, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:59:24.210790 8104 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260924 04:59:24.211189 8104 webserver.cc:460] Webserver started at http://127.0.98.2:40327/ using document root <none> and password file <none>
I20260924 04:59:24.211390 8104 fs_manager.cc:359] Metadata directory not provided
I20260924 04:59:24.211444 8104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:59:24.211571 8104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:59:24.212587 8104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225929848825-392-0/minicluster-data/ts-1/data/instance:
uuid: "b1756f97ef304ef1ad449fbf0d69c3e0"
format_stamp: "Formatted at 2026-09-24 04:59:24 on dist-test-slave-70st"
server_key: "83fd8e057ff549948304be46a9959b27"
server_key_iv: "935b0aaed4832ce8bcb2cb52856f723b"
server_key_version: "encryptionkey@0"
I20260924 04:59:24.212936 8104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225929848825-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b1756f97ef304ef1ad449fbf0d69c3e0"
format_stamp: "Formatted at 2026-09-24 04:59:24 on dist-test-slave-70st"
server_key: "83fd8e057ff549948304be46a9959b27"
server_key_iv: "935b0aaed4832ce8bcb2cb52856f723b"
server_key_version: "encryptionkey@0"
I20260924 04:59:24.214226 8104 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 04:59:24.214871 8119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:24.215035 8104 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 04:59:24.215116 8104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225929848825-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225929848825-392-0/minicluster-data/ts-1/wal
uuid: "b1756f97ef304ef1ad449fbf0d69c3e0"
format_stamp: "Formatted at 2026-09-24 04:59:24 on dist-test-slave-70st"
server_key: "83fd8e057ff549948304be46a9959b27"
server_key_iv: "935b0aaed4832ce8bcb2cb52856f723b"
server_key_version: "encryptionkey@0"
I20260924 04:59:24.215175 8104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225929848825-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225929848825-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225929848825-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:59:24.238704 8104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:59:24.240909 8104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:59:24.241030 8104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:59:24.241565 8104 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:59:24.241611 8104 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:24.241638 8104 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:59:24.241655 8104 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:24.248162 8104 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41169
I20260924 04:59:24.248209 8234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41169 every 8 connection(s)
Sep 24 04:59:24 dist-test-slave-70st krb5kdc[7847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225964, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 04:59:24.248561 8104 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225929848825-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:59:24.251221 7942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:24.241979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60063 (local address 127.0.98.62:44523)
0924 04:59:24.242019 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:24.242021 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:24.242270 (+ 249us) server_negotiation.cc:407] Connection header received
0924 04:59:24.242771 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:24.242772 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:24.242787 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:24.242854 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:24.243607 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:24.244362 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:24.245915 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:24.246281 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:24.246362 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:24.248946 (+ 2584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:24.248948 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:24.248948 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:59:24.248955 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:24.250136 (+ 1181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:24.250511 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:24.250512 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:24.250512 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:24.250559 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:24.250721 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:24.250721 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:24.250721 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:24.250839 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:24.250900 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:24.251099 (+ 199us) server_negotiation.cc:299] Negotiation successful
0924 04:59:24.251159 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 04:59:24.251430 8128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:24.241994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44523 (local address 127.0.98.2:60063)
0924 04:59:24.242140 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:24.242175 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:24.242678 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:24.242919 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:24.242922 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:24.243227 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:24.243513 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:24.243520 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:24.244465 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:24.244466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:24.245794 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:24.245798 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:24.246391 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:24.246392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:24.246506 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:24.247068 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:24.247079 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:24.248803 (+ 1724us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:59:24.250201 (+ 1398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:24.250207 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:24.250209 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:24.250453 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:24.250597 (+ 144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:24.250597 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:24.250598 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:24.250683 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:24.250885 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:59:24.250888 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:59:24.251000 (+ 112us) client_negotiation.cc:769] Sending connection context
0924 04:59:24.251081 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 04:59:24.251199 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":75,"threads_started":1}
I20260924 04:59:24.251842 8235 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44523
I20260924 04:59:24.251921 8235 heartbeater.cc:464] Registering TS with master...
I20260924 04:59:24.252089 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 8104
I20260924 04:59:24.252105 8235 heartbeater.cc:511] Master 127.0.98.62:44523 requested a full tablet report, sending...
I20260924 04:59:24.252193 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225929848825-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:59:24.252393 392 external_mini_cluster.cc:1428] Setting key a9d7a42f55df63bea92e946c83bfb10d
I20260924 04:59:24.252480 7892 ts_manager.cc:194] Registered new tserver with Master: b1756f97ef304ef1ad449fbf0d69c3e0 (127.0.98.2:41169)
I20260924 04:59:24.252888 7892 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60063
I20260924 04:59:24.258517 8241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:24.252874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44523 (local address 127.0.98.2:34123)
0924 04:59:24.253198 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:24.253208 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:24.253281 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:24.253455 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:24.253456 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:24.253636 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:24.253695 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:24.253697 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:24.254342 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:24.254343 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:24.254900 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:24.254901 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:24.255229 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:24.255230 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:24.255287 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:24.255674 (+ 387us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:24.255679 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:24.256059 (+ 380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:59:24.257622 (+ 1563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:24.257623 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:24.257623 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:24.257785 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:24.257974 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:24.257975 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:24.257975 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:24.258020 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:24.258292 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:59:24.258293 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:59:24.258324 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 04:59:24.258371 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 04:59:24.258442 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":129,"threads_started":1}
I20260924 04:59:24.258553 7942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:24.252926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34123 (local address 127.0.98.62:44523)
0924 04:59:24.252961 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:24.252963 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:24.253257 (+ 294us) server_negotiation.cc:407] Connection header received
0924 04:59:24.253345 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:24.253346 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:24.253355 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:24.253381 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:24.253791 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:24.254255 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:24.254973 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:24.255171 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:24.255225 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:24.256129 (+ 904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:24.256130 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:24.256131 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:59:24.256137 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:24.257549 (+ 1412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:24.257853 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:24.257853 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:24.257854 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:24.257919 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:24.258056 (+ 137us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:24.258056 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:24.258057 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:24.258237 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:24.258305 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:24.258397 (+ 92us) server_negotiation.cc:299] Negotiation successful
0924 04:59:24.258467 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225929848825-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225929848825-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260924 04:59:24.276737 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225929848825-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225929848825-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225929848825-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225929848825-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44523
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225929848825-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:59:24.354727 8244 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:59:24.354936 8244 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:59:24.354980 8244 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:59:24.356132 8244 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:59:24.356173 8244 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:59:24.356192 8244 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 04:59:24.356208 8244 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:59:24.357671 8244 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225929848825-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225929848825-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225929848825-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225929848825-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:44523
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8244
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225929848825-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 04:59:24.357892 8244 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:59:24.358091 8244 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:59:24.362941 8244 server_base.cc:1034] running on GCE node
W20260924 04:59:24.362903 8250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:59:24.362875 8249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:59:24.363116 8252 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:59:24.363366 8244 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:59:24.363638 8244 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:59:24.364774 8244 hybrid_clock.cc:648] HybridClock initialized: now 1790225964364752 us; error 36 us; skew 500 ppm
Sep 24 04:59:24 dist-test-slave-70st krb5kdc[7847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225964, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:59:24.367147 8244 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260924 04:59:24.367547 8244 webserver.cc:460] Webserver started at http://127.0.98.3:39289/ using document root <none> and password file <none>
I20260924 04:59:24.367733 8244 fs_manager.cc:359] Metadata directory not provided
I20260924 04:59:24.367777 8244 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:59:24.367902 8244 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:59:24.368933 8244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225929848825-392-0/minicluster-data/ts-2/data/instance:
uuid: "737c56f6ae6643f69f44ff6f056b27a3"
format_stamp: "Formatted at 2026-09-24 04:59:24 on dist-test-slave-70st"
server_key: "ab52ea44c732857c36757c758e792f0c"
server_key_iv: "b0035188c3a4248a7e2e4a6f09bc4349"
server_key_version: "encryptionkey@0"
I20260924 04:59:24.369257 8244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225929848825-392-0/minicluster-data/ts-2/wal/instance:
uuid: "737c56f6ae6643f69f44ff6f056b27a3"
format_stamp: "Formatted at 2026-09-24 04:59:24 on dist-test-slave-70st"
server_key: "ab52ea44c732857c36757c758e792f0c"
server_key_iv: "b0035188c3a4248a7e2e4a6f09bc4349"
server_key_version: "encryptionkey@0"
I20260924 04:59:24.370524 8244 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 04:59:24.371178 8259 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:24.371333 8244 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 04:59:24.371410 8244 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225929848825-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225929848825-392-0/minicluster-data/ts-2/wal
uuid: "737c56f6ae6643f69f44ff6f056b27a3"
format_stamp: "Formatted at 2026-09-24 04:59:24 on dist-test-slave-70st"
server_key: "ab52ea44c732857c36757c758e792f0c"
server_key_iv: "b0035188c3a4248a7e2e4a6f09bc4349"
server_key_version: "encryptionkey@0"
I20260924 04:59:24.371470 8244 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225929848825-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225929848825-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225929848825-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:59:24.387513 8244 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:59:24.389703 8244 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:59:24.389835 8244 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:59:24.390399 8244 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:59:24.390447 8244 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:24.390478 8244 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:59:24.390504 8244 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 04:59:24 dist-test-slave-70st krb5kdc[7847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225964, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 04:59:24.397621 8244 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37019
I20260924 04:59:24.397971 8244 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225929848825-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:59:24.397981 8374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37019 every 8 connection(s)
I20260924 04:59:24.399885 7942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:24.390864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40081 (local address 127.0.98.62:44523)
0924 04:59:24.390925 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:24.390927 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:24.391127 (+ 200us) server_negotiation.cc:407] Connection header received
0924 04:59:24.391628 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:24.391629 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:24.391642 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:24.391692 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:24.392516 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:24.393266 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:24.394479 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:24.394771 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:24.394857 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:24.397220 (+ 2363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:24.397222 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:24.397222 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:59:24.397230 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:24.398663 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:24.399068 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:24.399070 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:24.399070 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:24.399128 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:24.399312 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:24.399313 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:24.399313 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:24.399417 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:24.399463 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:24.399738 (+ 275us) server_negotiation.cc:299] Negotiation successful
0924 04:59:24.399811 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 04:59:24.400053 8268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:24.390842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44523 (local address 127.0.98.3:40081)
0924 04:59:24.390983 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:24.391021 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:24.391532 (+ 511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:24.391765 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:24.391768 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:24.392074 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:24.392390 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:24.392396 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:24.393364 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:24.393365 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:24.394409 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:24.394412 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:24.394896 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:24.394897 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:24.395096 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:24.395656 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:24.395665 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:24.397125 (+ 1460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:59:24.398730 (+ 1605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:24.398733 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:24.398735 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:24.398984 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:24.399186 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:24.399187 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:24.399187 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:24.399275 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:24.399494 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:59:24.399499 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:59:24.399623 (+ 124us) client_negotiation.cc:769] Sending connection context
0924 04:59:24.399720 (+ 97us) client_negotiation.cc:240] Negotiation successful
0924 04:59:24.399850 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":77,"threads_started":1}
I20260924 04:59:24.400550 8375 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44523
I20260924 04:59:24.400632 8375 heartbeater.cc:464] Registering TS with master...
I20260924 04:59:24.400804 8375 heartbeater.cc:511] Master 127.0.98.62:44523 requested a full tablet report, sending...
I20260924 04:59:24.401142 7891 ts_manager.cc:194] Registered new tserver with Master: 737c56f6ae6643f69f44ff6f056b27a3 (127.0.98.3:37019)
I20260924 04:59:24.401337 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 8244
I20260924 04:59:24.401467 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225929848825-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:59:24.401633 7891 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40081
I20260924 04:59:24.401659 392 external_mini_cluster.cc:1428] Setting key 8178c06eed18af561c5f565fa4530526
I20260924 04:59:24.402428 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:59:24.407778 7942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:24.401914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41167 (local address 127.0.98.62:44523)
0924 04:59:24.401950 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:24.401952 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:24.402150 (+ 198us) server_negotiation.cc:407] Connection header received
0924 04:59:24.402204 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:24.402205 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:24.402214 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:24.402263 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:24.402632 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:24.403322 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:24.404183 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:24.404437 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:24.404477 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:24.405500 (+ 1023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:24.405501 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:24.405502 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:59:24.405509 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:24.406813 (+ 1304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:24.407114 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:24.407115 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:24.407115 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:24.407162 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:24.407323 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:24.407324 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:24.407324 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:24.407423 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:24.407476 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:24.407613 (+ 137us) server_negotiation.cc:299] Negotiation successful
0924 04:59:24.407682 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 04:59:24.407794 8381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:24.401881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44523 (local address 127.0.98.3:41167)
0924 04:59:24.402098 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:24.402108 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:24.402161 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:24.402267 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:24.402268 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:24.402493 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:24.402561 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:24.402563 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:24.403397 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:24.403398 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:24.404084 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:24.404086 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:24.404551 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:24.404552 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:24.404610 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:24.405089 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:24.405093 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:24.405390 (+ 297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:59:24.406893 (+ 1503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:24.406894 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:24.406895 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:24.407055 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:24.407215 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:24.407216 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:24.407216 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:24.407271 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:24.407475 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:59:24.407475 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:59:24.407517 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 04:59:24.407586 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 04:59:24.407697 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":71,"threads_started":1}
Sep 24 04:59:24 dist-test-slave-70st krb5kdc[7847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225964, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 04:59:24 dist-test-slave-70st krb5kdc[7847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:59:24 dist-test-slave-70st krb5kdc[7847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 04:59:24.423108 7942 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:59:24.417915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49170 (local address 127.0.98.62:44523)
0924 04:59:24.417969 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:24.417972 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:24.417982 (+ 10us) server_negotiation.cc:407] Connection header received
0924 04:59:24.418042 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:24.418043 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:24.418052 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:24.418097 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:24.418712 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:24.419241 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:24.419943 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:24.420141 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:24.420194 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:24.423032 (+ 2838us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49170: BlockingRecv error: recv got EOF from 127.0.0.1:49170 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Sep 24 04:59:24 dist-test-slave-70st krb5kdc[7847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225964, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 04:59:24.430737 7942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:24.423934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49180 (local address 127.0.98.62:44523)
0924 04:59:24.423995 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:24.423997 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:24.424075 (+ 78us) server_negotiation.cc:407] Connection header received
0924 04:59:24.424152 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:24.424153 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:24.424163 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:24.424207 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:24.424674 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:24.425293 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:24.425998 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:24.426194 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:24.426252 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:24.428304 (+ 2052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:24.428306 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:24.428306 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:59:24.428313 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:24.429686 (+ 1373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:24.430014 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:24.430015 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:24.430015 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:24.430068 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:24.430240 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:24.430241 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:24.430241 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:24.430351 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:24.430445 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:24.430554 (+ 109us) server_negotiation.cc:299] Negotiation successful
0924 04:59:24.430629 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 04:59:24.432286 7891 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49180:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:59:24.433902 7891 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:59:24.440306 8397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:24.436049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34321 (local address 127.0.0.1:57468)
0924 04:59:24.436540 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:24.436547 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:24.436593 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:24.436756 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:24.436760 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:24.436762 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:59:24.436837 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:24.436839 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:24.437819 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:24.437820 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:24.439063 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:24.439065 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:24.439949 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:24.439950 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:24.440031 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:24.440048 (+ 17us) client_negotiation.cc:769] Sending connection context
0924 04:59:24.440114 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 04:59:24.440173 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"mutex_wait_us":2,"thread_start_us":86,"threads_started":1}
I20260924 04:59:24.440310 8398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:24.436045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37019 (local address 127.0.0.1:33324)
0924 04:59:24.436189 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:24.436220 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:24.436280 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:24.436861 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:24.436862 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:24.436864 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:59:24.436942 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:24.436944 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:24.438192 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:24.438193 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:24.439207 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:24.439209 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:24.440030 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:24.440031 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:24.440099 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:24.440103 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 04:59:24.440130 (+ 27us) client_negotiation.cc:240] Negotiation successful
0924 04:59:24.440175 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"mutex_wait_us":6,"thread_start_us":60,"threads_started":1}
I20260924 04:59:24.440477 8401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:24.436361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33324 (local address 127.0.98.3:37019)
0924 04:59:24.436711 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:24.436714 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:24.436723 (+ 9us) server_negotiation.cc:407] Connection header received
0924 04:59:24.436756 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:24.436758 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:24.436771 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:24.437013 (+ 242us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:59:24.437070 (+ 57us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:24.438113 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:24.439272 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:24.439940 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:24.440222 (+ 282us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:24.440307 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:24.440332 (+ 25us) server_negotiation.cc:299] Negotiation successful
0924 04:59:24.440371 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":103,"threads_started":1}
I20260924 04:59:24.441123 8400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:24.436318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57468 (local address 127.0.98.1:34321)
0924 04:59:24.436476 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:24.436481 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:24.436608 (+ 127us) server_negotiation.cc:407] Connection header received
0924 04:59:24.436655 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:24.436658 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:24.436673 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:24.436740 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:59:24.436936 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:24.437743 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:24.439135 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:24.439852 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:24.439940 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:24.440023 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:24.440962 (+ 939us) server_negotiation.cc:299] Negotiation successful
0924 04:59:24.441026 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":84,"threads_started":1}
I20260924 04:59:24.441183 8303 tablet_service.cc:1467] Processing CreateTablet for tablet 205d348ed51e4d379f1fc83115e8f43f (DEFAULT_TABLE table=test-table [id=58bc2d8bc85848ac8854a857ed6b6bfe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:59:24.441491 8399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:24.436049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41169 (local address 127.0.0.1:48420)
0924 04:59:24.436295 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:24.436343 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:24.436394 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:24.436862 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:24.436863 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:24.436864 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:59:24.436942 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:24.436945 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:24.439272 (+ 2327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:24.439272 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:24.440056 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:24.440057 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:24.441244 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:24.441245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:24.441305 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:24.441309 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 04:59:24.441359 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 04:59:24.441397 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"mutex_wait_us":22,"thread_start_us":87,"threads_started":1}
I20260924 04:59:24.441615 8303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 205d348ed51e4d379f1fc83115e8f43f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:59:24.441713 8402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:24.436520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48420 (local address 127.0.98.2:41169)
0924 04:59:24.436653 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:24.436657 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:24.436667 (+ 10us) server_negotiation.cc:407] Connection header received
0924 04:59:24.436705 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:24.436708 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:24.436723 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:24.436801 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:59:24.437016 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:24.438110 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:24.440132 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:24.440784 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:24.440847 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:24.440921 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:24.441587 (+ 666us) server_negotiation.cc:299] Negotiation successful
0924 04:59:24.441624 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":82,"threads_started":1}
I20260924 04:59:24.441761 8029 tablet_service.cc:1467] Processing CreateTablet for tablet 205d348ed51e4d379f1fc83115e8f43f (DEFAULT_TABLE table=test-table [id=58bc2d8bc85848ac8854a857ed6b6bfe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:59:24.442108 8029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 205d348ed51e4d379f1fc83115e8f43f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:59:24.442318 8169 tablet_service.cc:1467] Processing CreateTablet for tablet 205d348ed51e4d379f1fc83115e8f43f (DEFAULT_TABLE table=test-table [id=58bc2d8bc85848ac8854a857ed6b6bfe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:59:24.442613 8169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 205d348ed51e4d379f1fc83115e8f43f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:59:24.444327 8403 tablet_bootstrap.cc:492] T 205d348ed51e4d379f1fc83115e8f43f P 737c56f6ae6643f69f44ff6f056b27a3: Bootstrap starting.
I20260924 04:59:24.444867 8403 tablet_bootstrap.cc:654] T 205d348ed51e4d379f1fc83115e8f43f P 737c56f6ae6643f69f44ff6f056b27a3: Neither blocks nor log segments found. Creating new log.
I20260924 04:59:24.445104 8403 log.cc:824] T 205d348ed51e4d379f1fc83115e8f43f P 737c56f6ae6643f69f44ff6f056b27a3: Log is configured to *not* fsync() on all Append() calls
I20260924 04:59:24.445801 8403 tablet_bootstrap.cc:492] T 205d348ed51e4d379f1fc83115e8f43f P 737c56f6ae6643f69f44ff6f056b27a3: No bootstrap required, opened a new log
I20260924 04:59:24.445863 8403 ts_tablet_manager.cc:1397] T 205d348ed51e4d379f1fc83115e8f43f P 737c56f6ae6643f69f44ff6f056b27a3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 04:59:24.446082 8406 tablet_bootstrap.cc:492] T 205d348ed51e4d379f1fc83115e8f43f P b1756f97ef304ef1ad449fbf0d69c3e0: Bootstrap starting.
I20260924 04:59:24.446301 8405 tablet_bootstrap.cc:492] T 205d348ed51e4d379f1fc83115e8f43f P 00a93275756742d3b9fa9c06d2bd8da4: Bootstrap starting.
I20260924 04:59:24.446707 8406 tablet_bootstrap.cc:654] T 205d348ed51e4d379f1fc83115e8f43f P b1756f97ef304ef1ad449fbf0d69c3e0: Neither blocks nor log segments found. Creating new log.
I20260924 04:59:24.446785 8405 tablet_bootstrap.cc:654] T 205d348ed51e4d379f1fc83115e8f43f P 00a93275756742d3b9fa9c06d2bd8da4: Neither blocks nor log segments found. Creating new log.
I20260924 04:59:24.447017 8406 log.cc:824] T 205d348ed51e4d379f1fc83115e8f43f P b1756f97ef304ef1ad449fbf0d69c3e0: Log is configured to *not* fsync() on all Append() calls
I20260924 04:59:24.447050 8405 log.cc:824] T 205d348ed51e4d379f1fc83115e8f43f P 00a93275756742d3b9fa9c06d2bd8da4: Log is configured to *not* fsync() on all Append() calls
I20260924 04:59:24.447690 8405 tablet_bootstrap.cc:492] T 205d348ed51e4d379f1fc83115e8f43f P 00a93275756742d3b9fa9c06d2bd8da4: No bootstrap required, opened a new log
I20260924 04:59:24.447690 8406 tablet_bootstrap.cc:492] T 205d348ed51e4d379f1fc83115e8f43f P b1756f97ef304ef1ad449fbf0d69c3e0: No bootstrap required, opened a new log
I20260924 04:59:24.447752 8405 ts_tablet_manager.cc:1397] T 205d348ed51e4d379f1fc83115e8f43f P 00a93275756742d3b9fa9c06d2bd8da4: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:59:24.447752 8406 ts_tablet_manager.cc:1397] T 205d348ed51e4d379f1fc83115e8f43f P b1756f97ef304ef1ad449fbf0d69c3e0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 04:59:24.447687 8403 raft_consensus.cc:348] T 205d348ed51e4d379f1fc83115e8f43f P 737c56f6ae6643f69f44ff6f056b27a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "737c56f6ae6643f69f44ff6f056b27a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37019 } } peers { permanent_uuid: "00a93275756742d3b9fa9c06d2bd8da4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34321 } } peers { permanent_uuid: "b1756f97ef304ef1ad449fbf0d69c3e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41169 } }
I20260924 04:59:24.447810 8403 raft_consensus.cc:374] T 205d348ed51e4d379f1fc83115e8f43f P 737c56f6ae6643f69f44ff6f056b27a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:59:24.447830 8403 raft_consensus.cc:729] T 205d348ed51e4d379f1fc83115e8f43f P 737c56f6ae6643f69f44ff6f056b27a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 737c56f6ae6643f69f44ff6f056b27a3, State: Initialized, Role: FOLLOWER
I20260924 04:59:24.447921 8403 consensus_queue.cc:260] T 205d348ed51e4d379f1fc83115e8f43f P 737c56f6ae6643f69f44ff6f056b27a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "737c56f6ae6643f69f44ff6f056b27a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37019 } } peers { permanent_uuid: "00a93275756742d3b9fa9c06d2bd8da4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34321 } } peers { permanent_uuid: "b1756f97ef304ef1ad449fbf0d69c3e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41169 } }
I20260924 04:59:24.448164 8375 heartbeater.cc:503] Master 127.0.98.62:44523 was elected leader, sending a full tablet report...
I20260924 04:59:24.448136 8403 ts_tablet_manager.cc:1428] T 205d348ed51e4d379f1fc83115e8f43f P 737c56f6ae6643f69f44ff6f056b27a3: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 04:59:24.449662 8406 raft_consensus.cc:348] T 205d348ed51e4d379f1fc83115e8f43f P b1756f97ef304ef1ad449fbf0d69c3e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "737c56f6ae6643f69f44ff6f056b27a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37019 } } peers { permanent_uuid: "00a93275756742d3b9fa9c06d2bd8da4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34321 } } peers { permanent_uuid: "b1756f97ef304ef1ad449fbf0d69c3e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41169 } }
I20260924 04:59:24.449817 8406 raft_consensus.cc:374] T 205d348ed51e4d379f1fc83115e8f43f P b1756f97ef304ef1ad449fbf0d69c3e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:59:24.449858 8406 raft_consensus.cc:729] T 205d348ed51e4d379f1fc83115e8f43f P b1756f97ef304ef1ad449fbf0d69c3e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1756f97ef304ef1ad449fbf0d69c3e0, State: Initialized, Role: FOLLOWER
I20260924 04:59:24.449961 8406 consensus_queue.cc:260] T 205d348ed51e4d379f1fc83115e8f43f P b1756f97ef304ef1ad449fbf0d69c3e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "737c56f6ae6643f69f44ff6f056b27a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37019 } } peers { permanent_uuid: "00a93275756742d3b9fa9c06d2bd8da4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34321 } } peers { permanent_uuid: "b1756f97ef304ef1ad449fbf0d69c3e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41169 } }
I20260924 04:59:24.450076 8405 raft_consensus.cc:348] T 205d348ed51e4d379f1fc83115e8f43f P 00a93275756742d3b9fa9c06d2bd8da4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "737c56f6ae6643f69f44ff6f056b27a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37019 } } peers { permanent_uuid: "00a93275756742d3b9fa9c06d2bd8da4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34321 } } peers { permanent_uuid: "b1756f97ef304ef1ad449fbf0d69c3e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41169 } }
I20260924 04:59:24.450212 8406 ts_tablet_manager.cc:1428] T 205d348ed51e4d379f1fc83115e8f43f P b1756f97ef304ef1ad449fbf0d69c3e0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 04:59:24.450215 8405 raft_consensus.cc:374] T 205d348ed51e4d379f1fc83115e8f43f P 00a93275756742d3b9fa9c06d2bd8da4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:59:24.450233 8235 heartbeater.cc:503] Master 127.0.98.62:44523 was elected leader, sending a full tablet report...
I20260924 04:59:24.450245 8405 raft_consensus.cc:729] T 205d348ed51e4d379f1fc83115e8f43f P 00a93275756742d3b9fa9c06d2bd8da4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a93275756742d3b9fa9c06d2bd8da4, State: Initialized, Role: FOLLOWER
I20260924 04:59:24.450341 8405 consensus_queue.cc:260] T 205d348ed51e4d379f1fc83115e8f43f P 00a93275756742d3b9fa9c06d2bd8da4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "737c56f6ae6643f69f44ff6f056b27a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37019 } } peers { permanent_uuid: "00a93275756742d3b9fa9c06d2bd8da4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34321 } } peers { permanent_uuid: "b1756f97ef304ef1ad449fbf0d69c3e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41169 } }
I20260924 04:59:24.450556 8405 ts_tablet_manager.cc:1428] T 205d348ed51e4d379f1fc83115e8f43f P 00a93275756742d3b9fa9c06d2bd8da4: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 04:59:24.450680 8095 heartbeater.cc:503] Master 127.0.98.62:44523 was elected leader, sending a full tablet report...
W20260924 04:59:24.499162 8236 tablet.cc:2369] T 205d348ed51e4d379f1fc83115e8f43f P b1756f97ef304ef1ad449fbf0d69c3e0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 04:59:24.586190 8096 tablet.cc:2369] T 205d348ed51e4d379f1fc83115e8f43f P 00a93275756742d3b9fa9c06d2bd8da4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 04:59:24.648638 8376 tablet.cc:2369] T 205d348ed51e4d379f1fc83115e8f43f P 737c56f6ae6643f69f44ff6f056b27a3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:59:24.742949 8410 raft_consensus.cc:482] T 205d348ed51e4d379f1fc83115e8f43f P b1756f97ef304ef1ad449fbf0d69c3e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:59:24.743076 8410 raft_consensus.cc:504] T 205d348ed51e4d379f1fc83115e8f43f P b1756f97ef304ef1ad449fbf0d69c3e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "737c56f6ae6643f69f44ff6f056b27a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37019 } } peers { permanent_uuid: "00a93275756742d3b9fa9c06d2bd8da4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34321 } } peers { permanent_uuid: "b1756f97ef304ef1ad449fbf0d69c3e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41169 } }
I20260924 04:59:24.743455 8410 leader_election.cc:288] T 205d348ed51e4d379f1fc83115e8f43f P b1756f97ef304ef1ad449fbf0d69c3e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 737c56f6ae6643f69f44ff6f056b27a3 (127.0.98.3:37019), 00a93275756742d3b9fa9c06d2bd8da4 (127.0.98.1:34321)
I20260924 04:59:24.747473 8400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:24.743726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53623 (local address 127.0.98.1:34321)
0924 04:59:24.743780 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:24.743783 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:24.743853 (+ 70us) server_negotiation.cc:407] Connection header received
0924 04:59:24.743964 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:24.743965 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:24.743979 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:24.744038 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:59:24.744263 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:24.745008 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:24.746535 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:24.747101 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:24.747178 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:24.747190 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:24.747354 (+ 164us) server_negotiation.cc:299] Negotiation successful
0924 04:59:24.747408 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 04:59:24.747474 8401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:24.743752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38879 (local address 127.0.98.3:37019)
0924 04:59:24.743793 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:24.743795 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:24.743921 (+ 126us) server_negotiation.cc:407] Connection header received
0924 04:59:24.743988 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:24.743989 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:24.744001 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:24.744046 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:59:24.744236 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:24.745011 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:24.746545 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:24.747101 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:24.747178 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:24.747193 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:24.747354 (+ 161us) server_negotiation.cc:299] Negotiation successful
0924 04:59:24.747397 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 04:59:24.747473 8412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:24.743681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37019 (local address 127.0.98.2:38879)
0924 04:59:24.743864 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:24.743873 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:24.743924 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:24.744056 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:24.744057 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:24.744058 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:59:24.744127 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:24.744128 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:24.745103 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:24.745103 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:24.746440 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:24.746441 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:24.747183 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:24.747184 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:24.747278 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:24.747283 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 04:59:24.747336 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 04:59:24.747399 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":123,"thread_start_us":60,"threads_started":1}
I20260924 04:59:24.747480 8128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:24.743681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34321 (local address 127.0.98.2:53623)
0924 04:59:24.743757 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:24.743770 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:24.743851 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:24.744048 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:24.744049 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:24.744050 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:59:24.744124 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:24.744126 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:24.745103 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:24.745103 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:24.746423 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:24.746427 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:24.747183 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:24.747184 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:24.747278 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:24.747282 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 04:59:24.747336 (+ 54us) client_negotiation.cc:240] Negotiation successful
0924 04:59:24.747378 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":63}
I20260924 04:59:24.747841 8320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "205d348ed51e4d379f1fc83115e8f43f" candidate_uuid: "b1756f97ef304ef1ad449fbf0d69c3e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "737c56f6ae6643f69f44ff6f056b27a3" is_pre_election: true
I20260924 04:59:24.747838 8049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "205d348ed51e4d379f1fc83115e8f43f" candidate_uuid: "b1756f97ef304ef1ad449fbf0d69c3e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a93275756742d3b9fa9c06d2bd8da4" is_pre_election: true
I20260924 04:59:24.747988 8320 raft_consensus.cc:2454] T 205d348ed51e4d379f1fc83115e8f43f P 737c56f6ae6643f69f44ff6f056b27a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b1756f97ef304ef1ad449fbf0d69c3e0 in term 0.
I20260924 04:59:24.747989 8049 raft_consensus.cc:2454] T 205d348ed51e4d379f1fc83115e8f43f P 00a93275756742d3b9fa9c06d2bd8da4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b1756f97ef304ef1ad449fbf0d69c3e0 in term 0.
I20260924 04:59:24.748205 8120 leader_election.cc:302] T 205d348ed51e4d379f1fc83115e8f43f P b1756f97ef304ef1ad449fbf0d69c3e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 737c56f6ae6643f69f44ff6f056b27a3, b1756f97ef304ef1ad449fbf0d69c3e0; no voters:
I20260924 04:59:24.748338 8410 raft_consensus.cc:2781] T 205d348ed51e4d379f1fc83115e8f43f P b1756f97ef304ef1ad449fbf0d69c3e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:59:24.748387 8410 raft_consensus.cc:482] T 205d348ed51e4d379f1fc83115e8f43f P b1756f97ef304ef1ad449fbf0d69c3e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:59:24.748410 8410 raft_consensus.cc:3037] T 205d348ed51e4d379f1fc83115e8f43f P b1756f97ef304ef1ad449fbf0d69c3e0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:59:24.749318 8410 raft_consensus.cc:504] T 205d348ed51e4d379f1fc83115e8f43f P b1756f97ef304ef1ad449fbf0d69c3e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "737c56f6ae6643f69f44ff6f056b27a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37019 } } peers { permanent_uuid: "00a93275756742d3b9fa9c06d2bd8da4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34321 } } peers { permanent_uuid: "b1756f97ef304ef1ad449fbf0d69c3e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41169 } }
I20260924 04:59:24.749481 8410 leader_election.cc:288] T 205d348ed51e4d379f1fc83115e8f43f P b1756f97ef304ef1ad449fbf0d69c3e0 [CANDIDATE]: Term 1 election: Requested vote from peers 737c56f6ae6643f69f44ff6f056b27a3 (127.0.98.3:37019), 00a93275756742d3b9fa9c06d2bd8da4 (127.0.98.1:34321)
I20260924 04:59:24.749627 8320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "205d348ed51e4d379f1fc83115e8f43f" candidate_uuid: "b1756f97ef304ef1ad449fbf0d69c3e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "737c56f6ae6643f69f44ff6f056b27a3"
I20260924 04:59:24.749663 8049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "205d348ed51e4d379f1fc83115e8f43f" candidate_uuid: "b1756f97ef304ef1ad449fbf0d69c3e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a93275756742d3b9fa9c06d2bd8da4"
I20260924 04:59:24.749702 8320 raft_consensus.cc:3037] T 205d348ed51e4d379f1fc83115e8f43f P 737c56f6ae6643f69f44ff6f056b27a3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:59:24.749750 8049 raft_consensus.cc:3037] T 205d348ed51e4d379f1fc83115e8f43f P 00a93275756742d3b9fa9c06d2bd8da4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:59:24.750350 8320 raft_consensus.cc:2454] T 205d348ed51e4d379f1fc83115e8f43f P 737c56f6ae6643f69f44ff6f056b27a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1756f97ef304ef1ad449fbf0d69c3e0 in term 1.
I20260924 04:59:24.750407 8049 raft_consensus.cc:2454] T 205d348ed51e4d379f1fc83115e8f43f P 00a93275756742d3b9fa9c06d2bd8da4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1756f97ef304ef1ad449fbf0d69c3e0 in term 1.
I20260924 04:59:24.750512 8120 leader_election.cc:302] T 205d348ed51e4d379f1fc83115e8f43f P b1756f97ef304ef1ad449fbf0d69c3e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 737c56f6ae6643f69f44ff6f056b27a3, b1756f97ef304ef1ad449fbf0d69c3e0; no voters:
I20260924 04:59:24.750617 8410 raft_consensus.cc:2781] T 205d348ed51e4d379f1fc83115e8f43f P b1756f97ef304ef1ad449fbf0d69c3e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:59:24.750761 8410 raft_consensus.cc:686] T 205d348ed51e4d379f1fc83115e8f43f P b1756f97ef304ef1ad449fbf0d69c3e0 [term 1 LEADER]: Becoming Leader. State: Replica: b1756f97ef304ef1ad449fbf0d69c3e0, State: Running, Role: LEADER
I20260924 04:59:24.750849 8410 consensus_queue.cc:237] T 205d348ed51e4d379f1fc83115e8f43f P b1756f97ef304ef1ad449fbf0d69c3e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "737c56f6ae6643f69f44ff6f056b27a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37019 } } peers { permanent_uuid: "00a93275756742d3b9fa9c06d2bd8da4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34321 } } peers { permanent_uuid: "b1756f97ef304ef1ad449fbf0d69c3e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41169 } }
I20260924 04:59:24.751590 7891 catalog_manager.cc:5495] T 205d348ed51e4d379f1fc83115e8f43f P b1756f97ef304ef1ad449fbf0d69c3e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to b1756f97ef304ef1ad449fbf0d69c3e0 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "b1756f97ef304ef1ad449fbf0d69c3e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "737c56f6ae6643f69f44ff6f056b27a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37019 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00a93275756742d3b9fa9c06d2bd8da4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b1756f97ef304ef1ad449fbf0d69c3e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41169 } health_report { overall_health: HEALTHY } } }
I20260924 04:59:24.793075 8402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:24.790869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48430 (local address 127.0.98.2:41169)
0924 04:59:24.790941 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:24.790943 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:24.790951 (+ 8us) server_negotiation.cc:407] Connection header received
0924 04:59:24.790988 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:24.790989 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:24.791008 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:24.791069 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:59:24.791203 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:24.791667 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:24.792383 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:24.792567 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:24.792620 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:24.792771 (+ 151us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:59:24.792868 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:59:24.792942 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:24.792977 (+ 35us) server_negotiation.cc:299] Negotiation successful
0924 04:59:24.792997 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 04:59:24.794924 8049 raft_consensus.cc:1261] T 205d348ed51e4d379f1fc83115e8f43f P 00a93275756742d3b9fa9c06d2bd8da4 [term 1 FOLLOWER]: Refusing update from remote peer b1756f97ef304ef1ad449fbf0d69c3e0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:59:24.794919 8320 raft_consensus.cc:1261] T 205d348ed51e4d379f1fc83115e8f43f P 737c56f6ae6643f69f44ff6f056b27a3 [term 1 FOLLOWER]: Refusing update from remote peer b1756f97ef304ef1ad449fbf0d69c3e0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:59:24.795267 8410 consensus_queue.cc:1035] T 205d348ed51e4d379f1fc83115e8f43f P b1756f97ef304ef1ad449fbf0d69c3e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a93275756742d3b9fa9c06d2bd8da4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:59:24.795403 8413 consensus_queue.cc:1035] T 205d348ed51e4d379f1fc83115e8f43f P b1756f97ef304ef1ad449fbf0d69c3e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "737c56f6ae6643f69f44ff6f056b27a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:59:24.798067 8418 mvcc.cc:205] Tried to move back new op lower bound from 7332765551797137408 to 7332765551619940352. Current Snapshot: MvccSnapshot[applied={T|T < 7332765551797137408}]
I20260924 04:59:24.798230 8417 mvcc.cc:205] Tried to move back new op lower bound from 7332765551797137408 to 7332765551619940352. Current Snapshot: MvccSnapshot[applied={T|T < 7332765551797137408}]
I20260924 04:59:24.799772 7892 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49180:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:59:24.799866 7892 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49180:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:59:24.800521 7892 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8d8dc2f2261142e59c7c7ca2b072e3fd: Sending DeleteTablet for 3 replicas of tablet 205d348ed51e4d379f1fc83115e8f43f
I20260924 04:59:24.800837 8029 tablet_service.cc:1514] Processing DeleteTablet for tablet 205d348ed51e4d379f1fc83115e8f43f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:59:24 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57468
I20260924 04:59:24.800858 8169 tablet_service.cc:1514] Processing DeleteTablet for tablet 205d348ed51e4d379f1fc83115e8f43f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:59:24 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48420
I20260924 04:59:24.800849 8303 tablet_service.cc:1514] Processing DeleteTablet for tablet 205d348ed51e4d379f1fc83115e8f43f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:59:24 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33324
I20260924 04:59:24.801020 8426 tablet_replica.cc:330] T 205d348ed51e4d379f1fc83115e8f43f P 00a93275756742d3b9fa9c06d2bd8da4: stopping tablet replica
I20260924 04:59:24.801107 8426 raft_consensus.cc:2229] T 205d348ed51e4d379f1fc83115e8f43f P 00a93275756742d3b9fa9c06d2bd8da4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:59:24.801196 8426 raft_consensus.cc:2258] T 205d348ed51e4d379f1fc83115e8f43f P 00a93275756742d3b9fa9c06d2bd8da4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:59:24.801247 7892 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49180:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:59:24.801383 7892 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:59:24.801733 8426 ts_tablet_manager.cc:1905] T 205d348ed51e4d379f1fc83115e8f43f P 00a93275756742d3b9fa9c06d2bd8da4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:59:24.802603 8029 tablet_service.cc:1467] Processing CreateTablet for tablet 4396f7f9f02d441b9d7a3d0cfe810f8a (DEFAULT_TABLE table=test-table [id=a46e98e792a94d6082a5e35b7350b5c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:59:24.802748 8169 tablet_service.cc:1467] Processing CreateTablet for tablet 4396f7f9f02d441b9d7a3d0cfe810f8a (DEFAULT_TABLE table=test-table [id=a46e98e792a94d6082a5e35b7350b5c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:59:24.802807 8029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4396f7f9f02d441b9d7a3d0cfe810f8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:59:24.802851 8169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4396f7f9f02d441b9d7a3d0cfe810f8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:59:24.803484 8428 tablet_replica.cc:330] T 205d348ed51e4d379f1fc83115e8f43f P 737c56f6ae6643f69f44ff6f056b27a3: stopping tablet replica
I20260924 04:59:24.803555 8428 raft_consensus.cc:2229] T 205d348ed51e4d379f1fc83115e8f43f P 737c56f6ae6643f69f44ff6f056b27a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:59:24.803637 8428 raft_consensus.cc:2258] T 205d348ed51e4d379f1fc83115e8f43f P 737c56f6ae6643f69f44ff6f056b27a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:59:24.803767 8427 tablet_replica.cc:330] T 205d348ed51e4d379f1fc83115e8f43f P b1756f97ef304ef1ad449fbf0d69c3e0: stopping tablet replica
I20260924 04:59:24.803830 8427 raft_consensus.cc:2229] T 205d348ed51e4d379f1fc83115e8f43f P b1756f97ef304ef1ad449fbf0d69c3e0 [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:59:24.803889 8427 raft_consensus.cc:2258] T 205d348ed51e4d379f1fc83115e8f43f P b1756f97ef304ef1ad449fbf0d69c3e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:59:24.804164 8406 tablet_bootstrap.cc:492] T 4396f7f9f02d441b9d7a3d0cfe810f8a P b1756f97ef304ef1ad449fbf0d69c3e0: Bootstrap starting.
I20260924 04:59:24.804402 8303 tablet_service.cc:1467] Processing CreateTablet for tablet 4396f7f9f02d441b9d7a3d0cfe810f8a (DEFAULT_TABLE table=test-table [id=a46e98e792a94d6082a5e35b7350b5c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:59:24.804500 8303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4396f7f9f02d441b9d7a3d0cfe810f8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:59:24.804536 8406 tablet_bootstrap.cc:654] T 4396f7f9f02d441b9d7a3d0cfe810f8a P b1756f97ef304ef1ad449fbf0d69c3e0: Neither blocks nor log segments found. Creating new log.
I20260924 04:59:24.805104 8406 tablet_bootstrap.cc:492] T 4396f7f9f02d441b9d7a3d0cfe810f8a P b1756f97ef304ef1ad449fbf0d69c3e0: No bootstrap required, opened a new log
I20260924 04:59:24.805138 8406 ts_tablet_manager.cc:1397] T 4396f7f9f02d441b9d7a3d0cfe810f8a P b1756f97ef304ef1ad449fbf0d69c3e0: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:59:24.805233 8406 raft_consensus.cc:348] T 4396f7f9f02d441b9d7a3d0cfe810f8a P b1756f97ef304ef1ad449fbf0d69c3e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a93275756742d3b9fa9c06d2bd8da4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34321 } } peers { permanent_uuid: "b1756f97ef304ef1ad449fbf0d69c3e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41169 } } peers { permanent_uuid: "737c56f6ae6643f69f44ff6f056b27a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37019 } }
I20260924 04:59:24.805274 8406 raft_consensus.cc:374] T 4396f7f9f02d441b9d7a3d0cfe810f8a P b1756f97ef304ef1ad449fbf0d69c3e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:59:24.805285 8406 raft_consensus.cc:729] T 4396f7f9f02d441b9d7a3d0cfe810f8a P b1756f97ef304ef1ad449fbf0d69c3e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1756f97ef304ef1ad449fbf0d69c3e0, State: Initialized, Role: FOLLOWER
I20260924 04:59:24.805359 8406 consensus_queue.cc:260] T 4396f7f9f02d441b9d7a3d0cfe810f8a P b1756f97ef304ef1ad449fbf0d69c3e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a93275756742d3b9fa9c06d2bd8da4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34321 } } peers { permanent_uuid: "b1756f97ef304ef1ad449fbf0d69c3e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41169 } } peers { permanent_uuid: "737c56f6ae6643f69f44ff6f056b27a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37019 } }
I20260924 04:59:24.805464 8406 ts_tablet_manager.cc:1428] T 4396f7f9f02d441b9d7a3d0cfe810f8a P b1756f97ef304ef1ad449fbf0d69c3e0: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260924 04:59:24.805655 8428 ts_tablet_manager.cc:1905] T 205d348ed51e4d379f1fc83115e8f43f P 737c56f6ae6643f69f44ff6f056b27a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:59:24.805778 8403 tablet_bootstrap.cc:492] T 4396f7f9f02d441b9d7a3d0cfe810f8a P 737c56f6ae6643f69f44ff6f056b27a3: Bootstrap starting.
I20260924 04:59:24.806239 8403 tablet_bootstrap.cc:654] T 4396f7f9f02d441b9d7a3d0cfe810f8a P 737c56f6ae6643f69f44ff6f056b27a3: Neither blocks nor log segments found. Creating new log.
I20260924 04:59:24.806821 8403 tablet_bootstrap.cc:492] T 4396f7f9f02d441b9d7a3d0cfe810f8a P 737c56f6ae6643f69f44ff6f056b27a3: No bootstrap required, opened a new log
I20260924 04:59:24.806876 8403 ts_tablet_manager.cc:1397] T 4396f7f9f02d441b9d7a3d0cfe810f8a P 737c56f6ae6643f69f44ff6f056b27a3: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:59:24.807008 8403 raft_consensus.cc:348] T 4396f7f9f02d441b9d7a3d0cfe810f8a P 737c56f6ae6643f69f44ff6f056b27a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a93275756742d3b9fa9c06d2bd8da4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34321 } } peers { permanent_uuid: "b1756f97ef304ef1ad449fbf0d69c3e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41169 } } peers { permanent_uuid: "737c56f6ae6643f69f44ff6f056b27a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37019 } }
I20260924 04:59:24.807056 8427 ts_tablet_manager.cc:1905] T 205d348ed51e4d379f1fc83115e8f43f P b1756f97ef304ef1ad449fbf0d69c3e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:59:24.807085 8403 raft_consensus.cc:374] T 4396f7f9f02d441b9d7a3d0cfe810f8a P 737c56f6ae6643f69f44ff6f056b27a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:59:24.807104 8403 raft_consensus.cc:729] T 4396f7f9f02d441b9d7a3d0cfe810f8a P 737c56f6ae6643f69f44ff6f056b27a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 737c56f6ae6643f69f44ff6f056b27a3, State: Initialized, Role: FOLLOWER
I20260924 04:59:24.807031 8426 ts_tablet_manager.cc:1918] T 205d348ed51e4d379f1fc83115e8f43f P 00a93275756742d3b9fa9c06d2bd8da4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:59:24.807152 8403 consensus_queue.cc:260] T 4396f7f9f02d441b9d7a3d0cfe810f8a P 737c56f6ae6643f69f44ff6f056b27a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a93275756742d3b9fa9c06d2bd8da4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34321 } } peers { permanent_uuid: "b1756f97ef304ef1ad449fbf0d69c3e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41169 } } peers { permanent_uuid: "737c56f6ae6643f69f44ff6f056b27a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37019 } }
I20260924 04:59:24.807242 8403 ts_tablet_manager.cc:1428] T 4396f7f9f02d441b9d7a3d0cfe810f8a P 737c56f6ae6643f69f44ff6f056b27a3: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:24.807322 8426 log.cc:1196] T 205d348ed51e4d379f1fc83115e8f43f P 00a93275756742d3b9fa9c06d2bd8da4: Deleting WAL directory at /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225929848825-392-0/minicluster-data/ts-0/wal/wals/205d348ed51e4d379f1fc83115e8f43f
I20260924 04:59:24.807628 8426 ts_tablet_manager.cc:1939] T 205d348ed51e4d379f1fc83115e8f43f P 00a93275756742d3b9fa9c06d2bd8da4: Deleting consensus metadata
I20260924 04:59:24.807879 8428 ts_tablet_manager.cc:1918] T 205d348ed51e4d379f1fc83115e8f43f P 737c56f6ae6643f69f44ff6f056b27a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:59:24.807946 8428 log.cc:1196] T 205d348ed51e4d379f1fc83115e8f43f P 737c56f6ae6643f69f44ff6f056b27a3: Deleting WAL directory at /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225929848825-392-0/minicluster-data/ts-2/wal/wals/205d348ed51e4d379f1fc83115e8f43f
I20260924 04:59:24.807989 8405 tablet_bootstrap.cc:492] T 4396f7f9f02d441b9d7a3d0cfe810f8a P 00a93275756742d3b9fa9c06d2bd8da4: Bootstrap starting.
I20260924 04:59:24.808176 8428 ts_tablet_manager.cc:1939] T 205d348ed51e4d379f1fc83115e8f43f P 737c56f6ae6643f69f44ff6f056b27a3: Deleting consensus metadata
I20260924 04:59:24.808353 8405 tablet_bootstrap.cc:654] T 4396f7f9f02d441b9d7a3d0cfe810f8a P 00a93275756742d3b9fa9c06d2bd8da4: Neither blocks nor log segments found. Creating new log.
I20260924 04:59:24.808544 7876 catalog_manager.cc:4841] TS 737c56f6ae6643f69f44ff6f056b27a3 (127.0.98.3:37019): tablet 205d348ed51e4d379f1fc83115e8f43f (table test-table [id=58bc2d8bc85848ac8854a857ed6b6bfe]) successfully deleted
I20260924 04:59:24.808758 7878 catalog_manager.cc:4841] TS 00a93275756742d3b9fa9c06d2bd8da4 (127.0.98.1:34321): tablet 205d348ed51e4d379f1fc83115e8f43f (table test-table [id=58bc2d8bc85848ac8854a857ed6b6bfe]) successfully deleted
I20260924 04:59:24.809316 8405 tablet_bootstrap.cc:492] T 4396f7f9f02d441b9d7a3d0cfe810f8a P 00a93275756742d3b9fa9c06d2bd8da4: No bootstrap required, opened a new log
I20260924 04:59:24.809369 8405 ts_tablet_manager.cc:1397] T 4396f7f9f02d441b9d7a3d0cfe810f8a P 00a93275756742d3b9fa9c06d2bd8da4: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:59:24.809660 8405 raft_consensus.cc:348] T 4396f7f9f02d441b9d7a3d0cfe810f8a P 00a93275756742d3b9fa9c06d2bd8da4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a93275756742d3b9fa9c06d2bd8da4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34321 } } peers { permanent_uuid: "b1756f97ef304ef1ad449fbf0d69c3e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41169 } } peers { permanent_uuid: "737c56f6ae6643f69f44ff6f056b27a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37019 } }
I20260924 04:59:24.809738 8405 raft_consensus.cc:374] T 4396f7f9f02d441b9d7a3d0cfe810f8a P 00a93275756742d3b9fa9c06d2bd8da4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:59:24.809764 8405 raft_consensus.cc:729] T 4396f7f9f02d441b9d7a3d0cfe810f8a P 00a93275756742d3b9fa9c06d2bd8da4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a93275756742d3b9fa9c06d2bd8da4, State: Initialized, Role: FOLLOWER
I20260924 04:59:24.809763 8427 ts_tablet_manager.cc:1918] T 205d348ed51e4d379f1fc83115e8f43f P b1756f97ef304ef1ad449fbf0d69c3e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:59:24.809815 8427 log.cc:1196] T 205d348ed51e4d379f1fc83115e8f43f P b1756f97ef304ef1ad449fbf0d69c3e0: Deleting WAL directory at /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225929848825-392-0/minicluster-data/ts-1/wal/wals/205d348ed51e4d379f1fc83115e8f43f
I20260924 04:59:24.809815 8405 consensus_queue.cc:260] T 4396f7f9f02d441b9d7a3d0cfe810f8a P 00a93275756742d3b9fa9c06d2bd8da4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a93275756742d3b9fa9c06d2bd8da4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34321 } } peers { permanent_uuid: "b1756f97ef304ef1ad449fbf0d69c3e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41169 } } peers { permanent_uuid: "737c56f6ae6643f69f44ff6f056b27a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37019 } }
I20260924 04:59:24.809907 8405 ts_tablet_manager.cc:1428] T 4396f7f9f02d441b9d7a3d0cfe810f8a P 00a93275756742d3b9fa9c06d2bd8da4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 04:59:24.810151 8427 ts_tablet_manager.cc:1939] T 205d348ed51e4d379f1fc83115e8f43f P b1756f97ef304ef1ad449fbf0d69c3e0: Deleting consensus metadata
I20260924 04:59:24.810531 7879 catalog_manager.cc:4841] TS b1756f97ef304ef1ad449fbf0d69c3e0 (127.0.98.2:41169): tablet 205d348ed51e4d379f1fc83115e8f43f (table test-table [id=58bc2d8bc85848ac8854a857ed6b6bfe]) successfully deleted
I20260924 04:59:25.073654 8409 raft_consensus.cc:482] T 4396f7f9f02d441b9d7a3d0cfe810f8a P 737c56f6ae6643f69f44ff6f056b27a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:59:25.073755 8409 raft_consensus.cc:504] T 4396f7f9f02d441b9d7a3d0cfe810f8a P 737c56f6ae6643f69f44ff6f056b27a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a93275756742d3b9fa9c06d2bd8da4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34321 } } peers { permanent_uuid: "b1756f97ef304ef1ad449fbf0d69c3e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41169 } } peers { permanent_uuid: "737c56f6ae6643f69f44ff6f056b27a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37019 } }
I20260924 04:59:25.074101 8409 leader_election.cc:288] T 4396f7f9f02d441b9d7a3d0cfe810f8a P 737c56f6ae6643f69f44ff6f056b27a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00a93275756742d3b9fa9c06d2bd8da4 (127.0.98.1:34321), b1756f97ef304ef1ad449fbf0d69c3e0 (127.0.98.2:41169)
I20260924 04:59:25.077054 8402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:25.074362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44177 (local address 127.0.98.2:41169)
0924 04:59:25.074455 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:25.074457 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:25.074506 (+ 49us) server_negotiation.cc:407] Connection header received
0924 04:59:25.074545 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:25.074546 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:25.074556 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:25.074618 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:59:25.074782 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:25.075376 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:25.076368 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:25.076740 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:25.076792 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:25.076804 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:25.076958 (+ 154us) server_negotiation.cc:299] Negotiation successful
0924 04:59:25.076988 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 04:59:25.077056 8400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:25.074356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33581 (local address 127.0.98.1:34321)
0924 04:59:25.074498 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:25.074500 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:25.074610 (+ 110us) server_negotiation.cc:407] Connection header received
0924 04:59:25.074674 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:25.074675 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:25.074685 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:25.074739 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:59:25.074912 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:25.075545 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:25.076418 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:25.076755 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:25.076816 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:25.076826 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:25.076953 (+ 127us) server_negotiation.cc:299] Negotiation successful
0924 04:59:25.076987 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260924 04:59:25.077081 8432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:25.074288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41169 (local address 127.0.98.3:44177)
0924 04:59:25.074422 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:25.074434 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:25.074501 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:25.074638 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:25.074639 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:25.074641 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:59:25.074716 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:25.074718 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:25.075447 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:25.075448 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:25.076279 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:25.076281 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:25.076813 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:25.076813 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:25.076889 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:25.076892 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 04:59:25.076939 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 04:59:25.076986 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"mutex_wait_us":4,"thread_start_us":58,"threads_started":1}
I20260924 04:59:25.077081 8433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:25.074304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34321 (local address 127.0.98.3:33581)
0924 04:59:25.074540 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:25.074550 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:25.074609 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:25.074745 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:25.074746 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:25.074748 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:59:25.074805 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:25.074807 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:25.075624 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:25.075625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:25.076332 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:25.076335 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:25.076823 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:25.076824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:25.076891 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:25.076894 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 04:59:25.076939 (+ 45us) client_negotiation.cc:240] Negotiation successful
0924 04:59:25.076980 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"mutex_wait_us":15,"thread_start_us":91,"threads_started":1}
I20260924 04:59:25.077319 8189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4396f7f9f02d441b9d7a3d0cfe810f8a" candidate_uuid: "737c56f6ae6643f69f44ff6f056b27a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1756f97ef304ef1ad449fbf0d69c3e0" is_pre_election: true
I20260924 04:59:25.077486 8189 raft_consensus.cc:2454] T 4396f7f9f02d441b9d7a3d0cfe810f8a P b1756f97ef304ef1ad449fbf0d69c3e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 737c56f6ae6643f69f44ff6f056b27a3 in term 0.
I20260924 04:59:25.077486 8049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4396f7f9f02d441b9d7a3d0cfe810f8a" candidate_uuid: "737c56f6ae6643f69f44ff6f056b27a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a93275756742d3b9fa9c06d2bd8da4" is_pre_election: true
I20260924 04:59:25.077564 8049 raft_consensus.cc:2454] T 4396f7f9f02d441b9d7a3d0cfe810f8a P 00a93275756742d3b9fa9c06d2bd8da4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 737c56f6ae6643f69f44ff6f056b27a3 in term 0.
I20260924 04:59:25.077716 8262 leader_election.cc:302] T 4396f7f9f02d441b9d7a3d0cfe810f8a P 737c56f6ae6643f69f44ff6f056b27a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a93275756742d3b9fa9c06d2bd8da4, 737c56f6ae6643f69f44ff6f056b27a3; no voters:
I20260924 04:59:25.077841 8409 raft_consensus.cc:2781] T 4396f7f9f02d441b9d7a3d0cfe810f8a P 737c56f6ae6643f69f44ff6f056b27a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:59:25.077879 8409 raft_consensus.cc:482] T 4396f7f9f02d441b9d7a3d0cfe810f8a P 737c56f6ae6643f69f44ff6f056b27a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:59:25.077899 8409 raft_consensus.cc:3037] T 4396f7f9f02d441b9d7a3d0cfe810f8a P 737c56f6ae6643f69f44ff6f056b27a3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:59:25.078526 8409 raft_consensus.cc:504] T 4396f7f9f02d441b9d7a3d0cfe810f8a P 737c56f6ae6643f69f44ff6f056b27a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a93275756742d3b9fa9c06d2bd8da4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34321 } } peers { permanent_uuid: "b1756f97ef304ef1ad449fbf0d69c3e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41169 } } peers { permanent_uuid: "737c56f6ae6643f69f44ff6f056b27a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37019 } }
I20260924 04:59:25.078636 8409 leader_election.cc:288] T 4396f7f9f02d441b9d7a3d0cfe810f8a P 737c56f6ae6643f69f44ff6f056b27a3 [CANDIDATE]: Term 1 election: Requested vote from peers 00a93275756742d3b9fa9c06d2bd8da4 (127.0.98.1:34321), b1756f97ef304ef1ad449fbf0d69c3e0 (127.0.98.2:41169)
I20260924 04:59:25.078781 8049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4396f7f9f02d441b9d7a3d0cfe810f8a" candidate_uuid: "737c56f6ae6643f69f44ff6f056b27a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a93275756742d3b9fa9c06d2bd8da4"
I20260924 04:59:25.078852 8049 raft_consensus.cc:3037] T 4396f7f9f02d441b9d7a3d0cfe810f8a P 00a93275756742d3b9fa9c06d2bd8da4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:59:25.079433 8049 raft_consensus.cc:2454] T 4396f7f9f02d441b9d7a3d0cfe810f8a P 00a93275756742d3b9fa9c06d2bd8da4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 737c56f6ae6643f69f44ff6f056b27a3 in term 1.
I20260924 04:59:25.079523 8189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4396f7f9f02d441b9d7a3d0cfe810f8a" candidate_uuid: "737c56f6ae6643f69f44ff6f056b27a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1756f97ef304ef1ad449fbf0d69c3e0"
I20260924 04:59:25.079591 8189 raft_consensus.cc:3037] T 4396f7f9f02d441b9d7a3d0cfe810f8a P b1756f97ef304ef1ad449fbf0d69c3e0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:59:25.079619 8262 leader_election.cc:302] T 4396f7f9f02d441b9d7a3d0cfe810f8a P 737c56f6ae6643f69f44ff6f056b27a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a93275756742d3b9fa9c06d2bd8da4, 737c56f6ae6643f69f44ff6f056b27a3; no voters:
I20260924 04:59:25.079710 8409 raft_consensus.cc:2781] T 4396f7f9f02d441b9d7a3d0cfe810f8a P 737c56f6ae6643f69f44ff6f056b27a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:59:25.079828 8409 raft_consensus.cc:686] T 4396f7f9f02d441b9d7a3d0cfe810f8a P 737c56f6ae6643f69f44ff6f056b27a3 [term 1 LEADER]: Becoming Leader. State: Replica: 737c56f6ae6643f69f44ff6f056b27a3, State: Running, Role: LEADER
I20260924 04:59:25.079895 8409 consensus_queue.cc:237] T 4396f7f9f02d441b9d7a3d0cfe810f8a P 737c56f6ae6643f69f44ff6f056b27a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a93275756742d3b9fa9c06d2bd8da4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34321 } } peers { permanent_uuid: "b1756f97ef304ef1ad449fbf0d69c3e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41169 } } peers { permanent_uuid: "737c56f6ae6643f69f44ff6f056b27a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37019 } }
I20260924 04:59:25.080349 8189 raft_consensus.cc:2454] T 4396f7f9f02d441b9d7a3d0cfe810f8a P b1756f97ef304ef1ad449fbf0d69c3e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 737c56f6ae6643f69f44ff6f056b27a3 in term 1.
I20260924 04:59:25.080556 7892 catalog_manager.cc:5495] T 4396f7f9f02d441b9d7a3d0cfe810f8a P 737c56f6ae6643f69f44ff6f056b27a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 737c56f6ae6643f69f44ff6f056b27a3 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "737c56f6ae6643f69f44ff6f056b27a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a93275756742d3b9fa9c06d2bd8da4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b1756f97ef304ef1ad449fbf0d69c3e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "737c56f6ae6643f69f44ff6f056b27a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37019 } health_report { overall_health: HEALTHY } } }
Sep 24 04:59:25 dist-test-slave-70st krb5kdc[7847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225963, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 04:59:25.114853 8441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:25.102060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44523 (local address 127.0.0.1:49186)
0924 04:59:25.102223 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:25.102235 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:25.102297 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:25.102441 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:25.102442 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:25.102725 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:25.102769 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:25.102771 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:25.103445 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:25.103446 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:25.103964 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:25.103965 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:25.104304 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:25.104304 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:25.104338 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:25.111034 (+ 6696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:25.111040 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:25.112455 (+ 1415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:59:25.113942 (+ 1487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:25.113949 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:25.113951 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:25.114137 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:25.114322 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:25.114322 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:25.114323 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:25.114367 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:25.114556 (+ 189us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:59:25.114559 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:59:25.114605 (+ 46us) client_negotiation.cc:769] Sending connection context
0924 04:59:25.114651 (+ 46us) client_negotiation.cc:240] Negotiation successful
0924 04:59:25.114745 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":92,"threads_started":1}
I20260924 04:59:25.114904 8442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:25.102128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49186 (local address 127.0.98.62:44523)
0924 04:59:25.102265 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:25.102267 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:25.102279 (+ 12us) server_negotiation.cc:407] Connection header received
0924 04:59:25.102361 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:25.102362 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:25.102374 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:25.102432 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:25.102842 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:25.103377 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:25.104050 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:25.104244 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:25.104304 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:25.112547 (+ 8243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:25.112548 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:25.112549 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:59:25.112556 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:25.113873 (+ 1317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:25.114197 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:25.114198 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:25.114199 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:25.114251 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:25.114414 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:25.114415 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:25.114415 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:25.114517 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:25.114560 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:25.114674 (+ 114us) server_negotiation.cc:299] Negotiation successful
0924 04:59:25.114797 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":109,"thread_start_us":73,"threads_started":1}
I20260924 04:59:25.116804 7892 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49186:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 04:59:25.116991 7892 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:59:25.118319 8303 tablet_service.cc:1467] Processing CreateTablet for tablet 7118776089954371b38c329c305133ac (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=af2ba24104bf4fd3a21f7cb15ef472ec]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 04:59:25.118323 8169 tablet_service.cc:1467] Processing CreateTablet for tablet 7118776089954371b38c329c305133ac (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=af2ba24104bf4fd3a21f7cb15ef472ec]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 04:59:25.118455 8303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7118776089954371b38c329c305133ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:59:25.118462 8169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7118776089954371b38c329c305133ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:59:25.119510 8406 tablet_bootstrap.cc:492] T 7118776089954371b38c329c305133ac P b1756f97ef304ef1ad449fbf0d69c3e0: Bootstrap starting.
I20260924 04:59:25.119560 8403 tablet_bootstrap.cc:492] T 7118776089954371b38c329c305133ac P 737c56f6ae6643f69f44ff6f056b27a3: Bootstrap starting.
I20260924 04:59:25.119591 8029 tablet_service.cc:1467] Processing CreateTablet for tablet 7118776089954371b38c329c305133ac (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=af2ba24104bf4fd3a21f7cb15ef472ec]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 04:59:25.119709 8029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7118776089954371b38c329c305133ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:59:25.119823 8406 tablet_bootstrap.cc:654] T 7118776089954371b38c329c305133ac P b1756f97ef304ef1ad449fbf0d69c3e0: Neither blocks nor log segments found. Creating new log.
I20260924 04:59:25.119889 8403 tablet_bootstrap.cc:654] T 7118776089954371b38c329c305133ac P 737c56f6ae6643f69f44ff6f056b27a3: Neither blocks nor log segments found. Creating new log.
I20260924 04:59:25.120352 8406 tablet_bootstrap.cc:492] T 7118776089954371b38c329c305133ac P b1756f97ef304ef1ad449fbf0d69c3e0: No bootstrap required, opened a new log
I20260924 04:59:25.120404 8406 ts_tablet_manager.cc:1397] T 7118776089954371b38c329c305133ac P b1756f97ef304ef1ad449fbf0d69c3e0: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260924 04:59:25.120512 8406 raft_consensus.cc:348] T 7118776089954371b38c329c305133ac P b1756f97ef304ef1ad449fbf0d69c3e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a93275756742d3b9fa9c06d2bd8da4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34321 } } peers { permanent_uuid: "b1756f97ef304ef1ad449fbf0d69c3e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41169 } } peers { permanent_uuid: "737c56f6ae6643f69f44ff6f056b27a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37019 } }
I20260924 04:59:25.120584 8406 raft_consensus.cc:374] T 7118776089954371b38c329c305133ac P b1756f97ef304ef1ad449fbf0d69c3e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:59:25.120600 8406 raft_consensus.cc:729] T 7118776089954371b38c329c305133ac P b1756f97ef304ef1ad449fbf0d69c3e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1756f97ef304ef1ad449fbf0d69c3e0, State: Initialized, Role: FOLLOWER
I20260924 04:59:25.120635 8406 consensus_queue.cc:260] T 7118776089954371b38c329c305133ac P b1756f97ef304ef1ad449fbf0d69c3e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a93275756742d3b9fa9c06d2bd8da4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34321 } } peers { permanent_uuid: "b1756f97ef304ef1ad449fbf0d69c3e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41169 } } peers { permanent_uuid: "737c56f6ae6643f69f44ff6f056b27a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37019 } }
I20260924 04:59:25.120720 8406 ts_tablet_manager.cc:1428] T 7118776089954371b38c329c305133ac P b1756f97ef304ef1ad449fbf0d69c3e0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:25.121042 8403 tablet_bootstrap.cc:492] T 7118776089954371b38c329c305133ac P 737c56f6ae6643f69f44ff6f056b27a3: No bootstrap required, opened a new log
I20260924 04:59:25.121085 8403 ts_tablet_manager.cc:1397] T 7118776089954371b38c329c305133ac P 737c56f6ae6643f69f44ff6f056b27a3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 04:59:25.121093 8405 tablet_bootstrap.cc:492] T 7118776089954371b38c329c305133ac P 00a93275756742d3b9fa9c06d2bd8da4: Bootstrap starting.
I20260924 04:59:25.121131 8423 tablet_replica.cc:439] T 7118776089954371b38c329c305133ac P b1756f97ef304ef1ad449fbf0d69c3e0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a93275756742d3b9fa9c06d2bd8da4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34321 } } peers { permanent_uuid: "b1756f97ef304ef1ad449fbf0d69c3e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41169 } } peers { permanent_uuid: "737c56f6ae6643f69f44ff6f056b27a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37019 } } }
I20260924 04:59:25.121210 8423 tablet_replica.cc:442] T 7118776089954371b38c329c305133ac P b1756f97ef304ef1ad449fbf0d69c3e0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:59:25.121212 8403 raft_consensus.cc:348] T 7118776089954371b38c329c305133ac P 737c56f6ae6643f69f44ff6f056b27a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a93275756742d3b9fa9c06d2bd8da4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34321 } } peers { permanent_uuid: "b1756f97ef304ef1ad449fbf0d69c3e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41169 } } peers { permanent_uuid: "737c56f6ae6643f69f44ff6f056b27a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37019 } }
I20260924 04:59:25.121274 8403 raft_consensus.cc:374] T 7118776089954371b38c329c305133ac P 737c56f6ae6643f69f44ff6f056b27a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:59:25.121299 8403 raft_consensus.cc:729] T 7118776089954371b38c329c305133ac P 737c56f6ae6643f69f44ff6f056b27a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 737c56f6ae6643f69f44ff6f056b27a3, State: Initialized, Role: FOLLOWER
I20260924 04:59:25.121330 8403 consensus_queue.cc:260] T 7118776089954371b38c329c305133ac P 737c56f6ae6643f69f44ff6f056b27a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a93275756742d3b9fa9c06d2bd8da4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34321 } } peers { permanent_uuid: "b1756f97ef304ef1ad449fbf0d69c3e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41169 } } peers { permanent_uuid: "737c56f6ae6643f69f44ff6f056b27a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37019 } }
I20260924 04:59:25.121456 8403 ts_tablet_manager.cc:1428] T 7118776089954371b38c329c305133ac P 737c56f6ae6643f69f44ff6f056b27a3: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 04:59:25.121464 8405 tablet_bootstrap.cc:654] T 7118776089954371b38c329c305133ac P 00a93275756742d3b9fa9c06d2bd8da4: Neither blocks nor log segments found. Creating new log.
I20260924 04:59:25.121567 8409 tablet_replica.cc:439] T 7118776089954371b38c329c305133ac P 737c56f6ae6643f69f44ff6f056b27a3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a93275756742d3b9fa9c06d2bd8da4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34321 } } peers { permanent_uuid: "b1756f97ef304ef1ad449fbf0d69c3e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41169 } } peers { permanent_uuid: "737c56f6ae6643f69f44ff6f056b27a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37019 } } }
I20260924 04:59:25.121636 8409 tablet_replica.cc:442] T 7118776089954371b38c329c305133ac P 737c56f6ae6643f69f44ff6f056b27a3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:59:25.122073 8405 tablet_bootstrap.cc:492] T 7118776089954371b38c329c305133ac P 00a93275756742d3b9fa9c06d2bd8da4: No bootstrap required, opened a new log
I20260924 04:59:25.122121 8405 ts_tablet_manager.cc:1397] T 7118776089954371b38c329c305133ac P 00a93275756742d3b9fa9c06d2bd8da4: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260924 04:59:25.122249 8405 raft_consensus.cc:348] T 7118776089954371b38c329c305133ac P 00a93275756742d3b9fa9c06d2bd8da4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a93275756742d3b9fa9c06d2bd8da4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34321 } } peers { permanent_uuid: "b1756f97ef304ef1ad449fbf0d69c3e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41169 } } peers { permanent_uuid: "737c56f6ae6643f69f44ff6f056b27a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37019 } }
I20260924 04:59:25.122318 8405 raft_consensus.cc:374] T 7118776089954371b38c329c305133ac P 00a93275756742d3b9fa9c06d2bd8da4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:59:25.122335 8405 raft_consensus.cc:729] T 7118776089954371b38c329c305133ac P 00a93275756742d3b9fa9c06d2bd8da4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a93275756742d3b9fa9c06d2bd8da4, State: Initialized, Role: FOLLOWER
I20260924 04:59:25.122390 8405 consensus_queue.cc:260] T 7118776089954371b38c329c305133ac P 00a93275756742d3b9fa9c06d2bd8da4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a93275756742d3b9fa9c06d2bd8da4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34321 } } peers { permanent_uuid: "b1756f97ef304ef1ad449fbf0d69c3e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41169 } } peers { permanent_uuid: "737c56f6ae6643f69f44ff6f056b27a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37019 } }
I20260924 04:59:25.122476 8405 ts_tablet_manager.cc:1428] T 7118776089954371b38c329c305133ac P 00a93275756742d3b9fa9c06d2bd8da4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:25.122532 8411 tablet_replica.cc:439] T 7118776089954371b38c329c305133ac P 00a93275756742d3b9fa9c06d2bd8da4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a93275756742d3b9fa9c06d2bd8da4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34321 } } peers { permanent_uuid: "b1756f97ef304ef1ad449fbf0d69c3e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41169 } } peers { permanent_uuid: "737c56f6ae6643f69f44ff6f056b27a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37019 } } }
I20260924 04:59:25.122644 8411 tablet_replica.cc:442] T 7118776089954371b38c329c305133ac P 00a93275756742d3b9fa9c06d2bd8da4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:59:25.266846 8409 raft_consensus.cc:482] T 7118776089954371b38c329c305133ac P 737c56f6ae6643f69f44ff6f056b27a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:59:25.266938 8409 raft_consensus.cc:504] T 7118776089954371b38c329c305133ac P 737c56f6ae6643f69f44ff6f056b27a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a93275756742d3b9fa9c06d2bd8da4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34321 } } peers { permanent_uuid: "b1756f97ef304ef1ad449fbf0d69c3e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41169 } } peers { permanent_uuid: "737c56f6ae6643f69f44ff6f056b27a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37019 } }
I20260924 04:59:25.267082 8409 leader_election.cc:288] T 7118776089954371b38c329c305133ac P 737c56f6ae6643f69f44ff6f056b27a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00a93275756742d3b9fa9c06d2bd8da4 (127.0.98.1:34321), b1756f97ef304ef1ad449fbf0d69c3e0 (127.0.98.2:41169)
I20260924 04:59:25.267280 8189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7118776089954371b38c329c305133ac" candidate_uuid: "737c56f6ae6643f69f44ff6f056b27a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1756f97ef304ef1ad449fbf0d69c3e0" is_pre_election: true
I20260924 04:59:25.267307 8049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7118776089954371b38c329c305133ac" candidate_uuid: "737c56f6ae6643f69f44ff6f056b27a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a93275756742d3b9fa9c06d2bd8da4" is_pre_election: true
I20260924 04:59:25.267367 8189 raft_consensus.cc:2454] T 7118776089954371b38c329c305133ac P b1756f97ef304ef1ad449fbf0d69c3e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 737c56f6ae6643f69f44ff6f056b27a3 in term 0.
I20260924 04:59:25.267402 8049 raft_consensus.cc:2454] T 7118776089954371b38c329c305133ac P 00a93275756742d3b9fa9c06d2bd8da4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 737c56f6ae6643f69f44ff6f056b27a3 in term 0.
I20260924 04:59:25.267527 8263 leader_election.cc:302] T 7118776089954371b38c329c305133ac P 737c56f6ae6643f69f44ff6f056b27a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 737c56f6ae6643f69f44ff6f056b27a3, b1756f97ef304ef1ad449fbf0d69c3e0; no voters:
I20260924 04:59:25.267622 8409 raft_consensus.cc:2781] T 7118776089954371b38c329c305133ac P 737c56f6ae6643f69f44ff6f056b27a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:59:25.267664 8409 raft_consensus.cc:482] T 7118776089954371b38c329c305133ac P 737c56f6ae6643f69f44ff6f056b27a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:59:25.267678 8409 raft_consensus.cc:3037] T 7118776089954371b38c329c305133ac P 737c56f6ae6643f69f44ff6f056b27a3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:59:25.268282 8409 raft_consensus.cc:504] T 7118776089954371b38c329c305133ac P 737c56f6ae6643f69f44ff6f056b27a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a93275756742d3b9fa9c06d2bd8da4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34321 } } peers { permanent_uuid: "b1756f97ef304ef1ad449fbf0d69c3e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41169 } } peers { permanent_uuid: "737c56f6ae6643f69f44ff6f056b27a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37019 } }
I20260924 04:59:25.268392 8409 leader_election.cc:288] T 7118776089954371b38c329c305133ac P 737c56f6ae6643f69f44ff6f056b27a3 [CANDIDATE]: Term 1 election: Requested vote from peers 00a93275756742d3b9fa9c06d2bd8da4 (127.0.98.1:34321), b1756f97ef304ef1ad449fbf0d69c3e0 (127.0.98.2:41169)
I20260924 04:59:25.268524 8049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7118776089954371b38c329c305133ac" candidate_uuid: "737c56f6ae6643f69f44ff6f056b27a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a93275756742d3b9fa9c06d2bd8da4"
I20260924 04:59:25.268564 8189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7118776089954371b38c329c305133ac" candidate_uuid: "737c56f6ae6643f69f44ff6f056b27a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1756f97ef304ef1ad449fbf0d69c3e0"
I20260924 04:59:25.268605 8049 raft_consensus.cc:3037] T 7118776089954371b38c329c305133ac P 00a93275756742d3b9fa9c06d2bd8da4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:59:25.268621 8189 raft_consensus.cc:3037] T 7118776089954371b38c329c305133ac P b1756f97ef304ef1ad449fbf0d69c3e0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:59:25.269177 8189 raft_consensus.cc:2454] T 7118776089954371b38c329c305133ac P b1756f97ef304ef1ad449fbf0d69c3e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 737c56f6ae6643f69f44ff6f056b27a3 in term 1.
I20260924 04:59:25.269186 8049 raft_consensus.cc:2454] T 7118776089954371b38c329c305133ac P 00a93275756742d3b9fa9c06d2bd8da4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 737c56f6ae6643f69f44ff6f056b27a3 in term 1.
I20260924 04:59:25.269341 8263 leader_election.cc:302] T 7118776089954371b38c329c305133ac P 737c56f6ae6643f69f44ff6f056b27a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 00a93275756742d3b9fa9c06d2bd8da4, 737c56f6ae6643f69f44ff6f056b27a3, b1756f97ef304ef1ad449fbf0d69c3e0; no voters:
I20260924 04:59:25.269492 8409 raft_consensus.cc:2781] T 7118776089954371b38c329c305133ac P 737c56f6ae6643f69f44ff6f056b27a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:59:25.269550 8409 raft_consensus.cc:686] T 7118776089954371b38c329c305133ac P 737c56f6ae6643f69f44ff6f056b27a3 [term 1 LEADER]: Becoming Leader. State: Replica: 737c56f6ae6643f69f44ff6f056b27a3, State: Running, Role: LEADER
I20260924 04:59:25.269600 8409 consensus_queue.cc:237] T 7118776089954371b38c329c305133ac P 737c56f6ae6643f69f44ff6f056b27a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a93275756742d3b9fa9c06d2bd8da4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34321 } } peers { permanent_uuid: "b1756f97ef304ef1ad449fbf0d69c3e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41169 } } peers { permanent_uuid: "737c56f6ae6643f69f44ff6f056b27a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37019 } }
I20260924 04:59:25.269882 8434 tablet_replica.cc:439] T 7118776089954371b38c329c305133ac P 737c56f6ae6643f69f44ff6f056b27a3: TxnStatusTablet state changed. Reason: New leader 737c56f6ae6643f69f44ff6f056b27a3. Latest consensus state: current_term: 1 leader_uuid: "737c56f6ae6643f69f44ff6f056b27a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a93275756742d3b9fa9c06d2bd8da4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34321 } } peers { permanent_uuid: "b1756f97ef304ef1ad449fbf0d69c3e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41169 } } peers { permanent_uuid: "737c56f6ae6643f69f44ff6f056b27a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37019 } } }
I20260924 04:59:25.269963 8434 tablet_replica.cc:442] T 7118776089954371b38c329c305133ac P 737c56f6ae6643f69f44ff6f056b27a3: This TxnStatusTablet replica's current role is: LEADER
I20260924 04:59:25.270149 8447 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 04:59:25.270236 8447 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 04:59:25.270184 7891 catalog_manager.cc:5495] T 7118776089954371b38c329c305133ac P 737c56f6ae6643f69f44ff6f056b27a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 737c56f6ae6643f69f44ff6f056b27a3 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "737c56f6ae6643f69f44ff6f056b27a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a93275756742d3b9fa9c06d2bd8da4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b1756f97ef304ef1ad449fbf0d69c3e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "737c56f6ae6643f69f44ff6f056b27a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37019 } health_report { overall_health: HEALTHY } } }
I20260924 04:59:25.303422 8450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:25.301005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33338 (local address 127.0.98.3:37019)
0924 04:59:25.301132 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:25.301134 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:25.301160 (+ 26us) server_negotiation.cc:407] Connection header received
0924 04:59:25.301198 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:25.301198 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:25.301218 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:25.301290 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:59:25.301399 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:25.301958 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:25.302684 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:25.302890 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:25.302944 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:25.303099 (+ 155us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:59:25.303198 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:59:25.303258 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:25.303302 (+ 44us) server_negotiation.cc:299] Negotiation successful
0924 04:59:25.303323 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":50,"threads_started":1}
I20260924 04:59:25.303412 8441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:25.300924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37019 (local address 127.0.0.1:33338)
0924 04:59:25.300987 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:25.300999 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:25.301052 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:25.301295 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:25.301296 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:25.301297 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 04:59:25.301348 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:25.301351 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:25.302029 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:25.302030 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:25.302610 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:25.302612 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:25.302948 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:25.302949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:25.303005 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:25.303021 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 04:59:25.303250 (+ 229us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 04:59:25.303258 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 04:59:25.303300 (+ 42us) client_negotiation.cc:240] Negotiation successful
0924 04:59:25.303344 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260924 04:59:25.304571 8049 raft_consensus.cc:1261] T 7118776089954371b38c329c305133ac P 00a93275756742d3b9fa9c06d2bd8da4 [term 1 FOLLOWER]: Refusing update from remote peer 737c56f6ae6643f69f44ff6f056b27a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:59:25.304837 8189 raft_consensus.cc:1261] T 7118776089954371b38c329c305133ac P b1756f97ef304ef1ad449fbf0d69c3e0 [term 1 FOLLOWER]: Refusing update from remote peer 737c56f6ae6643f69f44ff6f056b27a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:59:25.304832 8434 consensus_queue.cc:1035] T 7118776089954371b38c329c305133ac P 737c56f6ae6643f69f44ff6f056b27a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a93275756742d3b9fa9c06d2bd8da4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:59:25.305166 8434 consensus_queue.cc:1035] T 7118776089954371b38c329c305133ac P 737c56f6ae6643f69f44ff6f056b27a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1756f97ef304ef1ad449fbf0d69c3e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:59:25.305570 8411 tablet_replica.cc:439] T 7118776089954371b38c329c305133ac P 00a93275756742d3b9fa9c06d2bd8da4: TxnStatusTablet state changed. Reason: New leader 737c56f6ae6643f69f44ff6f056b27a3. Latest consensus state: current_term: 1 leader_uuid: "737c56f6ae6643f69f44ff6f056b27a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a93275756742d3b9fa9c06d2bd8da4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34321 } } peers { permanent_uuid: "b1756f97ef304ef1ad449fbf0d69c3e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41169 } } peers { permanent_uuid: "737c56f6ae6643f69f44ff6f056b27a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37019 } } }
I20260924 04:59:25.305656 8411 tablet_replica.cc:442] T 7118776089954371b38c329c305133ac P 00a93275756742d3b9fa9c06d2bd8da4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:59:25.305861 8423 tablet_replica.cc:439] T 7118776089954371b38c329c305133ac P b1756f97ef304ef1ad449fbf0d69c3e0: TxnStatusTablet state changed. Reason: New leader 737c56f6ae6643f69f44ff6f056b27a3. Latest consensus state: current_term: 1 leader_uuid: "737c56f6ae6643f69f44ff6f056b27a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a93275756742d3b9fa9c06d2bd8da4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34321 } } peers { permanent_uuid: "b1756f97ef304ef1ad449fbf0d69c3e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41169 } } peers { permanent_uuid: "737c56f6ae6643f69f44ff6f056b27a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37019 } } }
I20260924 04:59:25.305939 8423 tablet_replica.cc:442] T 7118776089954371b38c329c305133ac P b1756f97ef304ef1ad449fbf0d69c3e0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:59:25.306955 8411 tablet_replica.cc:439] T 7118776089954371b38c329c305133ac P 00a93275756742d3b9fa9c06d2bd8da4: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "737c56f6ae6643f69f44ff6f056b27a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a93275756742d3b9fa9c06d2bd8da4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34321 } } peers { permanent_uuid: "b1756f97ef304ef1ad449fbf0d69c3e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41169 } } peers { permanent_uuid: "737c56f6ae6643f69f44ff6f056b27a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37019 } } }
I20260924 04:59:25.307040 8411 tablet_replica.cc:442] T 7118776089954371b38c329c305133ac P 00a93275756742d3b9fa9c06d2bd8da4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:59:25.306989 8423 tablet_replica.cc:439] T 7118776089954371b38c329c305133ac P b1756f97ef304ef1ad449fbf0d69c3e0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "737c56f6ae6643f69f44ff6f056b27a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a93275756742d3b9fa9c06d2bd8da4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34321 } } peers { permanent_uuid: "b1756f97ef304ef1ad449fbf0d69c3e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41169 } } peers { permanent_uuid: "737c56f6ae6643f69f44ff6f056b27a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37019 } } }
I20260924 04:59:25.307066 8423 tablet_replica.cc:442] T 7118776089954371b38c329c305133ac P b1756f97ef304ef1ad449fbf0d69c3e0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 04:59:25.307355 8434 tablet_replica.cc:439] T 7118776089954371b38c329c305133ac P 737c56f6ae6643f69f44ff6f056b27a3: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "737c56f6ae6643f69f44ff6f056b27a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a93275756742d3b9fa9c06d2bd8da4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34321 } } peers { permanent_uuid: "b1756f97ef304ef1ad449fbf0d69c3e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41169 } } peers { permanent_uuid: "737c56f6ae6643f69f44ff6f056b27a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37019 } } }
I20260924 04:59:25.307440 8434 tablet_replica.cc:442] T 7118776089954371b38c329c305133ac P 737c56f6ae6643f69f44ff6f056b27a3: This TxnStatusTablet replica's current role is: LEADER
I20260924 04:59:25.307682 8454 mvcc.cc:205] Tried to move back new op lower bound from 7332765553885532160 to 7332765553744580608. Current Snapshot: MvccSnapshot[applied={T|T < 7332765553885532160}]
I20260924 04:59:25.308128 8457 tablet_replica.cc:439] T 7118776089954371b38c329c305133ac P 737c56f6ae6643f69f44ff6f056b27a3: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "737c56f6ae6643f69f44ff6f056b27a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a93275756742d3b9fa9c06d2bd8da4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34321 } } peers { permanent_uuid: "b1756f97ef304ef1ad449fbf0d69c3e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41169 } } peers { permanent_uuid: "737c56f6ae6643f69f44ff6f056b27a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37019 } } }
I20260924 04:59:25.308365 8457 tablet_replica.cc:442] T 7118776089954371b38c329c305133ac P 737c56f6ae6643f69f44ff6f056b27a3: This TxnStatusTablet replica's current role is: LEADER
I20260924 04:59:25.310883 8450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:25.308543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33354 (local address 127.0.98.3:37019)
0924 04:59:25.308640 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:25.308642 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:25.308687 (+ 45us) server_negotiation.cc:407] Connection header received
0924 04:59:25.308747 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:25.308747 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:25.308757 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:25.308799 (+ 42us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:59:25.308923 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:25.309470 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:25.310254 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:25.310453 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:25.310527 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:25.310661 (+ 134us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:59:25.310692 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:59:25.310735 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:25.310801 (+ 66us) server_negotiation.cc:299] Negotiation successful
0924 04:59:25.310820 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260924 04:59:25.315146 8450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:25.313001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37705 (local address 127.0.98.3:37019)
0924 04:59:25.313033 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:25.313035 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:25.313153 (+ 118us) server_negotiation.cc:407] Connection header received
0924 04:59:25.313205 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:25.313206 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:25.313215 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:25.313251 (+ 36us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:59:25.313358 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:25.313831 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:25.314539 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:25.314710 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:25.314759 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:25.314900 (+ 141us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:59:25.314928 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:59:25.314976 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:25.315056 (+ 80us) server_negotiation.cc:299] Negotiation successful
0924 04:59:25.315072 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 04:59:25.315327 8464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:25.312952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37019 (local address 127.0.98.3:37705)
0924 04:59:25.313097 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:25.313105 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:25.313156 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:25.313253 (+ 97us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:25.313254 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:25.313255 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 04:59:25.313313 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:25.313315 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:25.313894 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:25.313895 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:25.314472 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:25.314473 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:25.314770 (+ 297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:25.314771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:25.314825 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:25.314840 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 04:59:25.314981 (+ 141us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 04:59:25.314982 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 04:59:25.315044 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 04:59:25.315085 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":60,"threads_started":1}
I20260924 04:59:25.317381 8189 raft_consensus.cc:1261] T 4396f7f9f02d441b9d7a3d0cfe810f8a P b1756f97ef304ef1ad449fbf0d69c3e0 [term 1 FOLLOWER]: Refusing update from remote peer 737c56f6ae6643f69f44ff6f056b27a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:59:25.317461 8049 raft_consensus.cc:1261] T 4396f7f9f02d441b9d7a3d0cfe810f8a P 00a93275756742d3b9fa9c06d2bd8da4 [term 1 FOLLOWER]: Refusing update from remote peer 737c56f6ae6643f69f44ff6f056b27a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:59:25.317641 8409 consensus_queue.cc:1035] T 4396f7f9f02d441b9d7a3d0cfe810f8a P 737c56f6ae6643f69f44ff6f056b27a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1756f97ef304ef1ad449fbf0d69c3e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:59:25.317729 8457 consensus_queue.cc:1035] T 4396f7f9f02d441b9d7a3d0cfe810f8a P 737c56f6ae6643f69f44ff6f056b27a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a93275756742d3b9fa9c06d2bd8da4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:59:25.331179 7891 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49180:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:59:25.331315 7891 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49180:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:59:25.331892 7891 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8d8dc2f2261142e59c7c7ca2b072e3fd: Sending DeleteTablet for 3 replicas of tablet 4396f7f9f02d441b9d7a3d0cfe810f8a
I20260924 04:59:25.332111 8169 tablet_service.cc:1514] Processing DeleteTablet for tablet 4396f7f9f02d441b9d7a3d0cfe810f8a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:59:25 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48420
I20260924 04:59:25.332217 8029 tablet_service.cc:1514] Processing DeleteTablet for tablet 4396f7f9f02d441b9d7a3d0cfe810f8a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:59:25 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57468
I20260924 04:59:25.332290 8468 tablet_replica.cc:330] T 4396f7f9f02d441b9d7a3d0cfe810f8a P b1756f97ef304ef1ad449fbf0d69c3e0: stopping tablet replica
I20260924 04:59:25.332340 8468 raft_consensus.cc:2229] T 4396f7f9f02d441b9d7a3d0cfe810f8a P b1756f97ef304ef1ad449fbf0d69c3e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:59:25.332373 8468 raft_consensus.cc:2258] T 4396f7f9f02d441b9d7a3d0cfe810f8a P b1756f97ef304ef1ad449fbf0d69c3e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:59:25.332576 8468 ts_tablet_manager.cc:1905] T 4396f7f9f02d441b9d7a3d0cfe810f8a P b1756f97ef304ef1ad449fbf0d69c3e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:59:25.332594 8303 tablet_service.cc:1514] Processing DeleteTablet for tablet 4396f7f9f02d441b9d7a3d0cfe810f8a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:59:25 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33324
I20260924 04:59:25.332782 8470 tablet_replica.cc:330] T 4396f7f9f02d441b9d7a3d0cfe810f8a P 737c56f6ae6643f69f44ff6f056b27a3: stopping tablet replica
I20260924 04:59:25.332875 8470 raft_consensus.cc:2229] T 4396f7f9f02d441b9d7a3d0cfe810f8a P 737c56f6ae6643f69f44ff6f056b27a3 [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:59:25.332990 8470 raft_consensus.cc:2258] T 4396f7f9f02d441b9d7a3d0cfe810f8a P 737c56f6ae6643f69f44ff6f056b27a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:59:25.333253 8470 ts_tablet_manager.cc:1905] T 4396f7f9f02d441b9d7a3d0cfe810f8a P 737c56f6ae6643f69f44ff6f056b27a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:59:25.334766 8468 ts_tablet_manager.cc:1918] T 4396f7f9f02d441b9d7a3d0cfe810f8a P b1756f97ef304ef1ad449fbf0d69c3e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 04:59:25.334766 8470 ts_tablet_manager.cc:1918] T 4396f7f9f02d441b9d7a3d0cfe810f8a P 737c56f6ae6643f69f44ff6f056b27a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 04:59:25.334828 8468 log.cc:1196] T 4396f7f9f02d441b9d7a3d0cfe810f8a P b1756f97ef304ef1ad449fbf0d69c3e0: Deleting WAL directory at /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225929848825-392-0/minicluster-data/ts-1/wal/wals/4396f7f9f02d441b9d7a3d0cfe810f8a
I20260924 04:59:25.334931 8470 log.cc:1196] T 4396f7f9f02d441b9d7a3d0cfe810f8a P 737c56f6ae6643f69f44ff6f056b27a3: Deleting WAL directory at /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225929848825-392-0/minicluster-data/ts-2/wal/wals/4396f7f9f02d441b9d7a3d0cfe810f8a
I20260924 04:59:25.335059 8468 ts_tablet_manager.cc:1939] T 4396f7f9f02d441b9d7a3d0cfe810f8a P b1756f97ef304ef1ad449fbf0d69c3e0: Deleting consensus metadata
I20260924 04:59:25.335238 8470 ts_tablet_manager.cc:1939] T 4396f7f9f02d441b9d7a3d0cfe810f8a P 737c56f6ae6643f69f44ff6f056b27a3: Deleting consensus metadata
I20260924 04:59:25.335340 7879 catalog_manager.cc:4841] TS b1756f97ef304ef1ad449fbf0d69c3e0 (127.0.98.2:41169): tablet 4396f7f9f02d441b9d7a3d0cfe810f8a (table test-table [id=a46e98e792a94d6082a5e35b7350b5c5]) successfully deleted
I20260924 04:59:25.335608 7876 catalog_manager.cc:4841] TS 737c56f6ae6643f69f44ff6f056b27a3 (127.0.98.3:37019): tablet 4396f7f9f02d441b9d7a3d0cfe810f8a (table test-table [id=a46e98e792a94d6082a5e35b7350b5c5]) successfully deleted
I20260924 04:59:25.336215 8469 tablet_replica.cc:330] T 4396f7f9f02d441b9d7a3d0cfe810f8a P 00a93275756742d3b9fa9c06d2bd8da4: stopping tablet replica
I20260924 04:59:25.336280 8469 raft_consensus.cc:2229] T 4396f7f9f02d441b9d7a3d0cfe810f8a P 00a93275756742d3b9fa9c06d2bd8da4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:59:25.336320 8469 raft_consensus.cc:2258] T 4396f7f9f02d441b9d7a3d0cfe810f8a P 00a93275756742d3b9fa9c06d2bd8da4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:59:25.336551 8469 ts_tablet_manager.cc:1905] T 4396f7f9f02d441b9d7a3d0cfe810f8a P 00a93275756742d3b9fa9c06d2bd8da4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:59:25.338060 8469 ts_tablet_manager.cc:1918] T 4396f7f9f02d441b9d7a3d0cfe810f8a P 00a93275756742d3b9fa9c06d2bd8da4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 04:59:25.338116 8469 log.cc:1196] T 4396f7f9f02d441b9d7a3d0cfe810f8a P 00a93275756742d3b9fa9c06d2bd8da4: Deleting WAL directory at /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225929848825-392-0/minicluster-data/ts-0/wal/wals/4396f7f9f02d441b9d7a3d0cfe810f8a
I20260924 04:59:25.338303 8469 ts_tablet_manager.cc:1939] T 4396f7f9f02d441b9d7a3d0cfe810f8a P 00a93275756742d3b9fa9c06d2bd8da4: Deleting consensus metadata
I20260924 04:59:25.338725 7878 catalog_manager.cc:4841] TS 00a93275756742d3b9fa9c06d2bd8da4 (127.0.98.1:34321): tablet 4396f7f9f02d441b9d7a3d0cfe810f8a (table test-table [id=a46e98e792a94d6082a5e35b7350b5c5]) successfully deleted
Sep 24 04:59:25 dist-test-slave-70st krb5kdc[7847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225965, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 04:59:25 dist-test-slave-70st krb5kdc[7847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:59:25 dist-test-slave-70st krb5kdc[7847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 04:59:25.363394 8442 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:59:25.358011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49188 (local address 127.0.98.62:44523)
0924 04:59:25.358076 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:25.358079 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:25.358158 (+ 79us) server_negotiation.cc:407] Connection header received
0924 04:59:25.358205 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:25.358206 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:25.358217 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:25.358290 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:25.358842 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:25.359650 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:25.360427 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:25.360744 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:25.360815 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:25.363314 (+ 2499us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49188: BlockingRecv error: recv got EOF from 127.0.0.1:49188 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 24 04:59:25 dist-test-slave-70st krb5kdc[7847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225965, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 04:59:25.370903 8442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:25.364078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49198 (local address 127.0.98.62:44523)
0924 04:59:25.364126 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:25.364128 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:25.364158 (+ 30us) server_negotiation.cc:407] Connection header received
0924 04:59:25.364204 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:25.364205 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:25.364213 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:25.364276 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:25.364655 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:25.365363 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:25.366083 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:25.366350 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:25.366428 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:25.368513 (+ 2085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:25.368515 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:25.368515 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:59:25.368523 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:25.369931 (+ 1408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:25.370222 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:25.370223 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:25.370224 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:25.370280 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:25.370441 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:25.370442 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:25.370443 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:25.370561 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:25.370656 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:25.370745 (+ 89us) server_negotiation.cc:299] Negotiation successful
0924 04:59:25.370817 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 04:59:25.372247 7891 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49198:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:59:25.372422 7891 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:59:25.373725 8029 tablet_service.cc:1467] Processing CreateTablet for tablet 00ce7243801443899e317de79def1fd9 (DEFAULT_TABLE table=test-table [id=08c9e2a745954e848c72efdd8b2bd941]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:59:25.373857 8029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00ce7243801443899e317de79def1fd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:59:25.373926 8169 tablet_service.cc:1467] Processing CreateTablet for tablet 00ce7243801443899e317de79def1fd9 (DEFAULT_TABLE table=test-table [id=08c9e2a745954e848c72efdd8b2bd941]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:59:25.374053 8169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00ce7243801443899e317de79def1fd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:59:25.375157 8406 tablet_bootstrap.cc:492] T 00ce7243801443899e317de79def1fd9 P b1756f97ef304ef1ad449fbf0d69c3e0: Bootstrap starting.
I20260924 04:59:25.375233 8303 tablet_service.cc:1467] Processing CreateTablet for tablet 00ce7243801443899e317de79def1fd9 (DEFAULT_TABLE table=test-table [id=08c9e2a745954e848c72efdd8b2bd941]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:59:25.375340 8303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00ce7243801443899e317de79def1fd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:59:25.375466 8406 tablet_bootstrap.cc:654] T 00ce7243801443899e317de79def1fd9 P b1756f97ef304ef1ad449fbf0d69c3e0: Neither blocks nor log segments found. Creating new log.
I20260924 04:59:25.375931 8406 tablet_bootstrap.cc:492] T 00ce7243801443899e317de79def1fd9 P b1756f97ef304ef1ad449fbf0d69c3e0: No bootstrap required, opened a new log
I20260924 04:59:25.375973 8406 ts_tablet_manager.cc:1397] T 00ce7243801443899e317de79def1fd9 P b1756f97ef304ef1ad449fbf0d69c3e0: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260924 04:59:25.376078 8406 raft_consensus.cc:348] T 00ce7243801443899e317de79def1fd9 P b1756f97ef304ef1ad449fbf0d69c3e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a93275756742d3b9fa9c06d2bd8da4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34321 } } peers { permanent_uuid: "b1756f97ef304ef1ad449fbf0d69c3e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41169 } } peers { permanent_uuid: "737c56f6ae6643f69f44ff6f056b27a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37019 } }
I20260924 04:59:25.376137 8406 raft_consensus.cc:374] T 00ce7243801443899e317de79def1fd9 P b1756f97ef304ef1ad449fbf0d69c3e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:59:25.376155 8406 raft_consensus.cc:729] T 00ce7243801443899e317de79def1fd9 P b1756f97ef304ef1ad449fbf0d69c3e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1756f97ef304ef1ad449fbf0d69c3e0, State: Initialized, Role: FOLLOWER
I20260924 04:59:25.376196 8406 consensus_queue.cc:260] T 00ce7243801443899e317de79def1fd9 P b1756f97ef304ef1ad449fbf0d69c3e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a93275756742d3b9fa9c06d2bd8da4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34321 } } peers { permanent_uuid: "b1756f97ef304ef1ad449fbf0d69c3e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41169 } } peers { permanent_uuid: "737c56f6ae6643f69f44ff6f056b27a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37019 } }
I20260924 04:59:25.376286 8406 ts_tablet_manager.cc:1428] T 00ce7243801443899e317de79def1fd9 P b1756f97ef304ef1ad449fbf0d69c3e0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:25.376545 8405 tablet_bootstrap.cc:492] T 00ce7243801443899e317de79def1fd9 P 00a93275756742d3b9fa9c06d2bd8da4: Bootstrap starting.
I20260924 04:59:25.376927 8405 tablet_bootstrap.cc:654] T 00ce7243801443899e317de79def1fd9 P 00a93275756742d3b9fa9c06d2bd8da4: Neither blocks nor log segments found. Creating new log.
I20260924 04:59:25.377223 8403 tablet_bootstrap.cc:492] T 00ce7243801443899e317de79def1fd9 P 737c56f6ae6643f69f44ff6f056b27a3: Bootstrap starting.
I20260924 04:59:25.377642 8403 tablet_bootstrap.cc:654] T 00ce7243801443899e317de79def1fd9 P 737c56f6ae6643f69f44ff6f056b27a3: Neither blocks nor log segments found. Creating new log.
I20260924 04:59:25.378048 8405 tablet_bootstrap.cc:492] T 00ce7243801443899e317de79def1fd9 P 00a93275756742d3b9fa9c06d2bd8da4: No bootstrap required, opened a new log
I20260924 04:59:25.378096 8405 ts_tablet_manager.cc:1397] T 00ce7243801443899e317de79def1fd9 P 00a93275756742d3b9fa9c06d2bd8da4: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 04:59:25.378196 8403 tablet_bootstrap.cc:492] T 00ce7243801443899e317de79def1fd9 P 737c56f6ae6643f69f44ff6f056b27a3: No bootstrap required, opened a new log
I20260924 04:59:25.378242 8403 ts_tablet_manager.cc:1397] T 00ce7243801443899e317de79def1fd9 P 737c56f6ae6643f69f44ff6f056b27a3: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260924 04:59:25.378235 8405 raft_consensus.cc:348] T 00ce7243801443899e317de79def1fd9 P 00a93275756742d3b9fa9c06d2bd8da4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a93275756742d3b9fa9c06d2bd8da4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34321 } } peers { permanent_uuid: "b1756f97ef304ef1ad449fbf0d69c3e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41169 } } peers { permanent_uuid: "737c56f6ae6643f69f44ff6f056b27a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37019 } }
I20260924 04:59:25.378300 8405 raft_consensus.cc:374] T 00ce7243801443899e317de79def1fd9 P 00a93275756742d3b9fa9c06d2bd8da4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:59:25.378324 8405 raft_consensus.cc:729] T 00ce7243801443899e317de79def1fd9 P 00a93275756742d3b9fa9c06d2bd8da4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a93275756742d3b9fa9c06d2bd8da4, State: Initialized, Role: FOLLOWER
I20260924 04:59:25.378358 8405 consensus_queue.cc:260] T 00ce7243801443899e317de79def1fd9 P 00a93275756742d3b9fa9c06d2bd8da4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a93275756742d3b9fa9c06d2bd8da4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34321 } } peers { permanent_uuid: "b1756f97ef304ef1ad449fbf0d69c3e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41169 } } peers { permanent_uuid: "737c56f6ae6643f69f44ff6f056b27a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37019 } }
I20260924 04:59:25.378402 8403 raft_consensus.cc:348] T 00ce7243801443899e317de79def1fd9 P 737c56f6ae6643f69f44ff6f056b27a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a93275756742d3b9fa9c06d2bd8da4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34321 } } peers { permanent_uuid: "b1756f97ef304ef1ad449fbf0d69c3e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41169 } } peers { permanent_uuid: "737c56f6ae6643f69f44ff6f056b27a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37019 } }
I20260924 04:59:25.378451 8405 ts_tablet_manager.cc:1428] T 00ce7243801443899e317de79def1fd9 P 00a93275756742d3b9fa9c06d2bd8da4: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260924 04:59:25.378470 8403 raft_consensus.cc:374] T 00ce7243801443899e317de79def1fd9 P 737c56f6ae6643f69f44ff6f056b27a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:59:25.378494 8403 raft_consensus.cc:729] T 00ce7243801443899e317de79def1fd9 P 737c56f6ae6643f69f44ff6f056b27a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 737c56f6ae6643f69f44ff6f056b27a3, State: Initialized, Role: FOLLOWER
I20260924 04:59:25.378530 8403 consensus_queue.cc:260] T 00ce7243801443899e317de79def1fd9 P 737c56f6ae6643f69f44ff6f056b27a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a93275756742d3b9fa9c06d2bd8da4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34321 } } peers { permanent_uuid: "b1756f97ef304ef1ad449fbf0d69c3e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41169 } } peers { permanent_uuid: "737c56f6ae6643f69f44ff6f056b27a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37019 } }
I20260924 04:59:25.378607 8403 ts_tablet_manager.cc:1428] T 00ce7243801443899e317de79def1fd9 P 737c56f6ae6643f69f44ff6f056b27a3: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:25.387722 8411 raft_consensus.cc:482] T 00ce7243801443899e317de79def1fd9 P 00a93275756742d3b9fa9c06d2bd8da4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:59:25.387801 8411 raft_consensus.cc:504] T 00ce7243801443899e317de79def1fd9 P 00a93275756742d3b9fa9c06d2bd8da4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a93275756742d3b9fa9c06d2bd8da4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34321 } } peers { permanent_uuid: "b1756f97ef304ef1ad449fbf0d69c3e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41169 } } peers { permanent_uuid: "737c56f6ae6643f69f44ff6f056b27a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37019 } }
I20260924 04:59:25.388096 8411 leader_election.cc:288] T 00ce7243801443899e317de79def1fd9 P 00a93275756742d3b9fa9c06d2bd8da4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b1756f97ef304ef1ad449fbf0d69c3e0 (127.0.98.2:41169), 737c56f6ae6643f69f44ff6f056b27a3 (127.0.98.3:37019)
I20260924 04:59:25.391661 8402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:25.388255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42499 (local address 127.0.98.2:41169)
0924 04:59:25.388320 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:25.388322 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:25.388483 (+ 161us) server_negotiation.cc:407] Connection header received
0924 04:59:25.388591 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:25.388592 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:25.388605 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:25.388654 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:59:25.388857 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:25.389571 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:25.391001 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:25.391350 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:25.391406 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:25.391417 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:25.391566 (+ 149us) server_negotiation.cc:299] Negotiation successful
0924 04:59:25.391599 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 04:59:25.391676 8486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:25.388221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41169 (local address 127.0.98.1:42499)
0924 04:59:25.388426 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:25.388437 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:25.388508 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:25.388698 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:25.388699 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:25.388721 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:59:25.388794 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:25.388795 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:25.389672 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:25.389673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:25.390908 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:25.390911 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:25.391443 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:25.391444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:25.391510 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:25.391513 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 04:59:25.391557 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 04:59:25.391594 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":90,"threads_started":1}
I20260924 04:59:25.391830 8189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00ce7243801443899e317de79def1fd9" candidate_uuid: "00a93275756742d3b9fa9c06d2bd8da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1756f97ef304ef1ad449fbf0d69c3e0" is_pre_election: true
I20260924 04:59:25.391896 8189 raft_consensus.cc:2454] T 00ce7243801443899e317de79def1fd9 P b1756f97ef304ef1ad449fbf0d69c3e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00a93275756742d3b9fa9c06d2bd8da4 in term 0.
I20260924 04:59:25.392064 7983 leader_election.cc:302] T 00ce7243801443899e317de79def1fd9 P 00a93275756742d3b9fa9c06d2bd8da4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a93275756742d3b9fa9c06d2bd8da4, b1756f97ef304ef1ad449fbf0d69c3e0; no voters:
I20260924 04:59:25.392168 8411 raft_consensus.cc:2781] T 00ce7243801443899e317de79def1fd9 P 00a93275756742d3b9fa9c06d2bd8da4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:59:25.392190 8411 raft_consensus.cc:482] T 00ce7243801443899e317de79def1fd9 P 00a93275756742d3b9fa9c06d2bd8da4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:59:25.392206 8411 raft_consensus.cc:3037] T 00ce7243801443899e317de79def1fd9 P 00a93275756742d3b9fa9c06d2bd8da4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:59:25.392963 8411 raft_consensus.cc:504] T 00ce7243801443899e317de79def1fd9 P 00a93275756742d3b9fa9c06d2bd8da4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a93275756742d3b9fa9c06d2bd8da4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34321 } } peers { permanent_uuid: "b1756f97ef304ef1ad449fbf0d69c3e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41169 } } peers { permanent_uuid: "737c56f6ae6643f69f44ff6f056b27a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37019 } }
I20260924 04:59:25.393078 8411 leader_election.cc:288] T 00ce7243801443899e317de79def1fd9 P 00a93275756742d3b9fa9c06d2bd8da4 [CANDIDATE]: Term 1 election: Requested vote from peers b1756f97ef304ef1ad449fbf0d69c3e0 (127.0.98.2:41169), 737c56f6ae6643f69f44ff6f056b27a3 (127.0.98.3:37019)
I20260924 04:59:25.393281 8189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00ce7243801443899e317de79def1fd9" candidate_uuid: "00a93275756742d3b9fa9c06d2bd8da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1756f97ef304ef1ad449fbf0d69c3e0"
I20260924 04:59:25.393358 8189 raft_consensus.cc:3037] T 00ce7243801443899e317de79def1fd9 P b1756f97ef304ef1ad449fbf0d69c3e0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:59:25.393440 8450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:25.388438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34669 (local address 127.0.98.3:37019)
0924 04:59:25.388475 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:25.388477 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:25.388485 (+ 8us) server_negotiation.cc:407] Connection header received
0924 04:59:25.388556 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:25.388557 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:25.388575 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:25.388620 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:59:25.388769 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:25.389563 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:25.392505 (+ 2942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:25.393060 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:25.393108 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:25.393117 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:25.393318 (+ 201us) server_negotiation.cc:299] Negotiation successful
0924 04:59:25.393361 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 04:59:25.393458 8487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:25.388266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37019 (local address 127.0.98.1:34669)
0924 04:59:25.388442 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:25.388451 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:25.388496 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:25.388621 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:25.388622 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:25.388624 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:59:25.388695 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:25.388697 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:25.391006 (+ 2309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:25.391007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:25.392399 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:25.392402 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:25.393177 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:25.393178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:25.393252 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:25.393255 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 04:59:25.393309 (+ 54us) client_negotiation.cc:240] Negotiation successful
0924 04:59:25.393346 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":74,"threads_started":1}
I20260924 04:59:25.393980 8189 raft_consensus.cc:2454] T 00ce7243801443899e317de79def1fd9 P b1756f97ef304ef1ad449fbf0d69c3e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00a93275756742d3b9fa9c06d2bd8da4 in term 1.
I20260924 04:59:25.394160 7983 leader_election.cc:302] T 00ce7243801443899e317de79def1fd9 P 00a93275756742d3b9fa9c06d2bd8da4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a93275756742d3b9fa9c06d2bd8da4, b1756f97ef304ef1ad449fbf0d69c3e0; no voters:
I20260924 04:59:25.394229 8320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00ce7243801443899e317de79def1fd9" candidate_uuid: "00a93275756742d3b9fa9c06d2bd8da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "737c56f6ae6643f69f44ff6f056b27a3" is_pre_election: true
I20260924 04:59:25.394249 8329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00ce7243801443899e317de79def1fd9" candidate_uuid: "00a93275756742d3b9fa9c06d2bd8da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "737c56f6ae6643f69f44ff6f056b27a3"
I20260924 04:59:25.394304 8320 raft_consensus.cc:2454] T 00ce7243801443899e317de79def1fd9 P 737c56f6ae6643f69f44ff6f056b27a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00a93275756742d3b9fa9c06d2bd8da4 in term 0.
I20260924 04:59:25.394378 8411 raft_consensus.cc:2781] T 00ce7243801443899e317de79def1fd9 P 00a93275756742d3b9fa9c06d2bd8da4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:59:25.394505 8411 raft_consensus.cc:686] T 00ce7243801443899e317de79def1fd9 P 00a93275756742d3b9fa9c06d2bd8da4 [term 1 LEADER]: Becoming Leader. State: Replica: 00a93275756742d3b9fa9c06d2bd8da4, State: Running, Role: LEADER
I20260924 04:59:25.394570 8411 consensus_queue.cc:237] T 00ce7243801443899e317de79def1fd9 P 00a93275756742d3b9fa9c06d2bd8da4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a93275756742d3b9fa9c06d2bd8da4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34321 } } peers { permanent_uuid: "b1756f97ef304ef1ad449fbf0d69c3e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41169 } } peers { permanent_uuid: "737c56f6ae6643f69f44ff6f056b27a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37019 } }
I20260924 04:59:25.395246 7891 catalog_manager.cc:5495] T 00ce7243801443899e317de79def1fd9 P 00a93275756742d3b9fa9c06d2bd8da4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 00a93275756742d3b9fa9c06d2bd8da4 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "00a93275756742d3b9fa9c06d2bd8da4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a93275756742d3b9fa9c06d2bd8da4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b1756f97ef304ef1ad449fbf0d69c3e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "737c56f6ae6643f69f44ff6f056b27a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37019 } health_report { overall_health: UNKNOWN } } }
I20260924 04:59:25.401943 8400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:25.399221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57472 (local address 127.0.98.1:34321)
0924 04:59:25.399271 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:25.399273 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:25.399280 (+ 7us) server_negotiation.cc:407] Connection header received
0924 04:59:25.399309 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:25.399310 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:25.399328 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:25.399397 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:59:25.399543 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:25.400212 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:25.401255 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:25.401485 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:25.401541 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:25.401695 (+ 154us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:59:25.401779 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:59:25.401835 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:25.401884 (+ 49us) server_negotiation.cc:299] Negotiation successful
0924 04:59:25.401898 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 04:59:25.402665 8329 raft_consensus.cc:3037] T 00ce7243801443899e317de79def1fd9 P 737c56f6ae6643f69f44ff6f056b27a3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:59:25.402693 8189 raft_consensus.cc:1261] T 00ce7243801443899e317de79def1fd9 P b1756f97ef304ef1ad449fbf0d69c3e0 [term 1 FOLLOWER]: Refusing update from remote peer 00a93275756742d3b9fa9c06d2bd8da4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:59:25.402872 8488 consensus_queue.cc:1035] T 00ce7243801443899e317de79def1fd9 P 00a93275756742d3b9fa9c06d2bd8da4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1756f97ef304ef1ad449fbf0d69c3e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:59:25.403509 8329 raft_consensus.cc:1261] T 00ce7243801443899e317de79def1fd9 P 737c56f6ae6643f69f44ff6f056b27a3 [term 1 FOLLOWER]: Refusing update from remote peer 00a93275756742d3b9fa9c06d2bd8da4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:59:25.403820 8488 consensus_queue.cc:1035] T 00ce7243801443899e317de79def1fd9 P 00a93275756742d3b9fa9c06d2bd8da4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "737c56f6ae6643f69f44ff6f056b27a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:59:25.406200 7892 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49198:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:59:25.406282 7892 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49198:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:59:25.406855 7892 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8d8dc2f2261142e59c7c7ca2b072e3fd: Sending DeleteTablet for 3 replicas of tablet 00ce7243801443899e317de79def1fd9
I20260924 04:59:25.407066 8029 tablet_service.cc:1514] Processing DeleteTablet for tablet 00ce7243801443899e317de79def1fd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:59:25 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57468
I20260924 04:59:25.407102 8303 tablet_service.cc:1514] Processing DeleteTablet for tablet 00ce7243801443899e317de79def1fd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:59:25 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33324
I20260924 04:59:25.407145 8469 tablet_replica.cc:330] T 00ce7243801443899e317de79def1fd9 P 00a93275756742d3b9fa9c06d2bd8da4: stopping tablet replica
I20260924 04:59:25.407176 8470 tablet_replica.cc:330] T 00ce7243801443899e317de79def1fd9 P 737c56f6ae6643f69f44ff6f056b27a3: stopping tablet replica
I20260924 04:59:25.407186 8469 raft_consensus.cc:2229] T 00ce7243801443899e317de79def1fd9 P 00a93275756742d3b9fa9c06d2bd8da4 [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:59:25.407215 8470 raft_consensus.cc:2229] T 00ce7243801443899e317de79def1fd9 P 737c56f6ae6643f69f44ff6f056b27a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:59:25.407238 8469 raft_consensus.cc:2258] T 00ce7243801443899e317de79def1fd9 P 00a93275756742d3b9fa9c06d2bd8da4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:59:25.407255 8470 raft_consensus.cc:2258] T 00ce7243801443899e317de79def1fd9 P 737c56f6ae6643f69f44ff6f056b27a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:59:25.407519 8469 ts_tablet_manager.cc:1905] T 00ce7243801443899e317de79def1fd9 P 00a93275756742d3b9fa9c06d2bd8da4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:59:25.407519 8470 ts_tablet_manager.cc:1905] T 00ce7243801443899e317de79def1fd9 P 737c56f6ae6643f69f44ff6f056b27a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:59:25.408954 8169 tablet_service.cc:1514] Processing DeleteTablet for tablet 00ce7243801443899e317de79def1fd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:59:25 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48420
I20260924 04:59:25.408981 8470 ts_tablet_manager.cc:1918] T 00ce7243801443899e317de79def1fd9 P 737c56f6ae6643f69f44ff6f056b27a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:59:25.409020 8470 log.cc:1196] T 00ce7243801443899e317de79def1fd9 P 737c56f6ae6643f69f44ff6f056b27a3: Deleting WAL directory at /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225929848825-392-0/minicluster-data/ts-2/wal/wals/00ce7243801443899e317de79def1fd9
I20260924 04:59:25.409209 8470 ts_tablet_manager.cc:1939] T 00ce7243801443899e317de79def1fd9 P 737c56f6ae6643f69f44ff6f056b27a3: Deleting consensus metadata
I20260924 04:59:25.409237 7892 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49198:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:59:25.409368 7892 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:59:25.409451 8468 tablet_replica.cc:330] T 00ce7243801443899e317de79def1fd9 P b1756f97ef304ef1ad449fbf0d69c3e0: stopping tablet replica
I20260924 04:59:25.409502 8468 raft_consensus.cc:2229] T 00ce7243801443899e317de79def1fd9 P b1756f97ef304ef1ad449fbf0d69c3e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:59:25.409545 8468 raft_consensus.cc:2258] T 00ce7243801443899e317de79def1fd9 P b1756f97ef304ef1ad449fbf0d69c3e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:59:25.409574 8469 ts_tablet_manager.cc:1918] T 00ce7243801443899e317de79def1fd9 P 00a93275756742d3b9fa9c06d2bd8da4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:59:25.409615 8469 log.cc:1196] T 00ce7243801443899e317de79def1fd9 P 00a93275756742d3b9fa9c06d2bd8da4: Deleting WAL directory at /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225929848825-392-0/minicluster-data/ts-0/wal/wals/00ce7243801443899e317de79def1fd9
I20260924 04:59:25.409675 7876 catalog_manager.cc:4841] TS 737c56f6ae6643f69f44ff6f056b27a3 (127.0.98.3:37019): tablet 00ce7243801443899e317de79def1fd9 (table test-table [id=08c9e2a745954e848c72efdd8b2bd941]) successfully deleted
I20260924 04:59:25.409799 8469 ts_tablet_manager.cc:1939] T 00ce7243801443899e317de79def1fd9 P 00a93275756742d3b9fa9c06d2bd8da4: Deleting consensus metadata
I20260924 04:59:25.410218 8468 ts_tablet_manager.cc:1905] T 00ce7243801443899e317de79def1fd9 P b1756f97ef304ef1ad449fbf0d69c3e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:59:25.410540 8303 tablet_service.cc:1467] Processing CreateTablet for tablet f49fdae751cb4280a642fdaff98b52e8 (DEFAULT_TABLE table=test-table [id=33762fe4e582447fb8a68d6ee8da2863]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:59:25.410630 8303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f49fdae751cb4280a642fdaff98b52e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:59:25.411564 8468 ts_tablet_manager.cc:1918] T 00ce7243801443899e317de79def1fd9 P b1756f97ef304ef1ad449fbf0d69c3e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:59:25.411608 8468 log.cc:1196] T 00ce7243801443899e317de79def1fd9 P b1756f97ef304ef1ad449fbf0d69c3e0: Deleting WAL directory at /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225929848825-392-0/minicluster-data/ts-1/wal/wals/00ce7243801443899e317de79def1fd9
I20260924 04:59:25.411630 8403 tablet_bootstrap.cc:492] T f49fdae751cb4280a642fdaff98b52e8 P 737c56f6ae6643f69f44ff6f056b27a3: Bootstrap starting.
I20260924 04:59:25.411800 8468 ts_tablet_manager.cc:1939] T 00ce7243801443899e317de79def1fd9 P b1756f97ef304ef1ad449fbf0d69c3e0: Deleting consensus metadata
I20260924 04:59:25.411882 8403 tablet_bootstrap.cc:654] T f49fdae751cb4280a642fdaff98b52e8 P 737c56f6ae6643f69f44ff6f056b27a3: Neither blocks nor log segments found. Creating new log.
I20260924 04:59:25.412389 8403 tablet_bootstrap.cc:492] T f49fdae751cb4280a642fdaff98b52e8 P 737c56f6ae6643f69f44ff6f056b27a3: No bootstrap required, opened a new log
I20260924 04:59:25.412429 8403 ts_tablet_manager.cc:1397] T f49fdae751cb4280a642fdaff98b52e8 P 737c56f6ae6643f69f44ff6f056b27a3: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260924 04:59:25.412521 8403 raft_consensus.cc:348] T f49fdae751cb4280a642fdaff98b52e8 P 737c56f6ae6643f69f44ff6f056b27a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a93275756742d3b9fa9c06d2bd8da4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34321 } } peers { permanent_uuid: "b1756f97ef304ef1ad449fbf0d69c3e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41169 } } peers { permanent_uuid: "737c56f6ae6643f69f44ff6f056b27a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37019 } }
I20260924 04:59:25.412585 8403 raft_consensus.cc:374] T f49fdae751cb4280a642fdaff98b52e8 P 737c56f6ae6643f69f44ff6f056b27a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:59:25.412606 8403 raft_consensus.cc:729] T f49fdae751cb4280a642fdaff98b52e8 P 737c56f6ae6643f69f44ff6f056b27a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 737c56f6ae6643f69f44ff6f056b27a3, State: Initialized, Role: FOLLOWER
I20260924 04:59:25.412642 8403 consensus_queue.cc:260] T f49fdae751cb4280a642fdaff98b52e8 P 737c56f6ae6643f69f44ff6f056b27a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a93275756742d3b9fa9c06d2bd8da4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34321 } } peers { permanent_uuid: "b1756f97ef304ef1ad449fbf0d69c3e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41169 } } peers { permanent_uuid: "737c56f6ae6643f69f44ff6f056b27a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37019 } }
I20260924 04:59:25.412688 8029 tablet_service.cc:1467] Processing CreateTablet for tablet f49fdae751cb4280a642fdaff98b52e8 (DEFAULT_TABLE table=test-table [id=33762fe4e582447fb8a68d6ee8da2863]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:59:25.412719 8403 ts_tablet_manager.cc:1428] T f49fdae751cb4280a642fdaff98b52e8 P 737c56f6ae6643f69f44ff6f056b27a3: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:25.412774 8029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f49fdae751cb4280a642fdaff98b52e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:59:25.412820 8169 tablet_service.cc:1467] Processing CreateTablet for tablet f49fdae751cb4280a642fdaff98b52e8 (DEFAULT_TABLE table=test-table [id=33762fe4e582447fb8a68d6ee8da2863]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:59:25.412902 8169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f49fdae751cb4280a642fdaff98b52e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:59:25.413870 8405 tablet_bootstrap.cc:492] T f49fdae751cb4280a642fdaff98b52e8 P 00a93275756742d3b9fa9c06d2bd8da4: Bootstrap starting.
I20260924 04:59:25.414238 8405 tablet_bootstrap.cc:654] T f49fdae751cb4280a642fdaff98b52e8 P 00a93275756742d3b9fa9c06d2bd8da4: Neither blocks nor log segments found. Creating new log.
I20260924 04:59:25.414425 8406 tablet_bootstrap.cc:492] T f49fdae751cb4280a642fdaff98b52e8 P b1756f97ef304ef1ad449fbf0d69c3e0: Bootstrap starting.
I20260924 04:59:25.414517 7878 catalog_manager.cc:4841] TS 00a93275756742d3b9fa9c06d2bd8da4 (127.0.98.1:34321): tablet 00ce7243801443899e317de79def1fd9 (table test-table [id=08c9e2a745954e848c72efdd8b2bd941]) successfully deleted
I20260924 04:59:25.414671 8406 tablet_bootstrap.cc:654] T f49fdae751cb4280a642fdaff98b52e8 P b1756f97ef304ef1ad449fbf0d69c3e0: Neither blocks nor log segments found. Creating new log.
I20260924 04:59:25.415086 7879 catalog_manager.cc:4841] TS b1756f97ef304ef1ad449fbf0d69c3e0 (127.0.98.2:41169): tablet 00ce7243801443899e317de79def1fd9 (table test-table [id=08c9e2a745954e848c72efdd8b2bd941]) successfully deleted
I20260924 04:59:25.417829 8405 tablet_bootstrap.cc:492] T f49fdae751cb4280a642fdaff98b52e8 P 00a93275756742d3b9fa9c06d2bd8da4: No bootstrap required, opened a new log
I20260924 04:59:25.417882 8405 ts_tablet_manager.cc:1397] T f49fdae751cb4280a642fdaff98b52e8 P 00a93275756742d3b9fa9c06d2bd8da4: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.000s
I20260924 04:59:25.417934 8406 tablet_bootstrap.cc:492] T f49fdae751cb4280a642fdaff98b52e8 P b1756f97ef304ef1ad449fbf0d69c3e0: No bootstrap required, opened a new log
I20260924 04:59:25.417979 8406 ts_tablet_manager.cc:1397] T f49fdae751cb4280a642fdaff98b52e8 P b1756f97ef304ef1ad449fbf0d69c3e0: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.000s
I20260924 04:59:25.418010 8405 raft_consensus.cc:348] T f49fdae751cb4280a642fdaff98b52e8 P 00a93275756742d3b9fa9c06d2bd8da4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a93275756742d3b9fa9c06d2bd8da4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34321 } } peers { permanent_uuid: "b1756f97ef304ef1ad449fbf0d69c3e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41169 } } peers { permanent_uuid: "737c56f6ae6643f69f44ff6f056b27a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37019 } }
I20260924 04:59:25.418093 8405 raft_consensus.cc:374] T f49fdae751cb4280a642fdaff98b52e8 P 00a93275756742d3b9fa9c06d2bd8da4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:59:25.418071 8406 raft_consensus.cc:348] T f49fdae751cb4280a642fdaff98b52e8 P b1756f97ef304ef1ad449fbf0d69c3e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a93275756742d3b9fa9c06d2bd8da4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34321 } } peers { permanent_uuid: "b1756f97ef304ef1ad449fbf0d69c3e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41169 } } peers { permanent_uuid: "737c56f6ae6643f69f44ff6f056b27a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37019 } }
I20260924 04:59:25.418113 8405 raft_consensus.cc:729] T f49fdae751cb4280a642fdaff98b52e8 P 00a93275756742d3b9fa9c06d2bd8da4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a93275756742d3b9fa9c06d2bd8da4, State: Initialized, Role: FOLLOWER
I20260924 04:59:25.418131 8406 raft_consensus.cc:374] T f49fdae751cb4280a642fdaff98b52e8 P b1756f97ef304ef1ad449fbf0d69c3e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:59:25.418150 8406 raft_consensus.cc:729] T f49fdae751cb4280a642fdaff98b52e8 P b1756f97ef304ef1ad449fbf0d69c3e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1756f97ef304ef1ad449fbf0d69c3e0, State: Initialized, Role: FOLLOWER
I20260924 04:59:25.418151 8405 consensus_queue.cc:260] T f49fdae751cb4280a642fdaff98b52e8 P 00a93275756742d3b9fa9c06d2bd8da4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a93275756742d3b9fa9c06d2bd8da4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34321 } } peers { permanent_uuid: "b1756f97ef304ef1ad449fbf0d69c3e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41169 } } peers { permanent_uuid: "737c56f6ae6643f69f44ff6f056b27a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37019 } }
I20260924 04:59:25.418190 8406 consensus_queue.cc:260] T f49fdae751cb4280a642fdaff98b52e8 P b1756f97ef304ef1ad449fbf0d69c3e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a93275756742d3b9fa9c06d2bd8da4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34321 } } peers { permanent_uuid: "b1756f97ef304ef1ad449fbf0d69c3e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41169 } } peers { permanent_uuid: "737c56f6ae6643f69f44ff6f056b27a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37019 } }
I20260924 04:59:25.418241 8405 ts_tablet_manager.cc:1428] T f49fdae751cb4280a642fdaff98b52e8 P 00a93275756742d3b9fa9c06d2bd8da4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:25.418264 8406 ts_tablet_manager.cc:1428] T f49fdae751cb4280a642fdaff98b52e8 P b1756f97ef304ef1ad449fbf0d69c3e0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:25.531849 8423 raft_consensus.cc:482] T f49fdae751cb4280a642fdaff98b52e8 P b1756f97ef304ef1ad449fbf0d69c3e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:59:25.531946 8423 raft_consensus.cc:504] T f49fdae751cb4280a642fdaff98b52e8 P b1756f97ef304ef1ad449fbf0d69c3e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a93275756742d3b9fa9c06d2bd8da4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34321 } } peers { permanent_uuid: "b1756f97ef304ef1ad449fbf0d69c3e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41169 } } peers { permanent_uuid: "737c56f6ae6643f69f44ff6f056b27a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37019 } }
I20260924 04:59:25.532097 8423 leader_election.cc:288] T f49fdae751cb4280a642fdaff98b52e8 P b1756f97ef304ef1ad449fbf0d69c3e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00a93275756742d3b9fa9c06d2bd8da4 (127.0.98.1:34321), 737c56f6ae6643f69f44ff6f056b27a3 (127.0.98.3:37019)
I20260924 04:59:25.532343 8329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49fdae751cb4280a642fdaff98b52e8" candidate_uuid: "b1756f97ef304ef1ad449fbf0d69c3e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "737c56f6ae6643f69f44ff6f056b27a3" is_pre_election: true
I20260924 04:59:25.532349 8049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49fdae751cb4280a642fdaff98b52e8" candidate_uuid: "b1756f97ef304ef1ad449fbf0d69c3e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a93275756742d3b9fa9c06d2bd8da4" is_pre_election: true
I20260924 04:59:25.532459 8049 raft_consensus.cc:2454] T f49fdae751cb4280a642fdaff98b52e8 P 00a93275756742d3b9fa9c06d2bd8da4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b1756f97ef304ef1ad449fbf0d69c3e0 in term 0.
I20260924 04:59:25.532459 8329 raft_consensus.cc:2454] T f49fdae751cb4280a642fdaff98b52e8 P 737c56f6ae6643f69f44ff6f056b27a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b1756f97ef304ef1ad449fbf0d69c3e0 in term 0.
I20260924 04:59:25.532610 8120 leader_election.cc:302] T f49fdae751cb4280a642fdaff98b52e8 P b1756f97ef304ef1ad449fbf0d69c3e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 737c56f6ae6643f69f44ff6f056b27a3, b1756f97ef304ef1ad449fbf0d69c3e0; no voters:
I20260924 04:59:25.532712 8423 raft_consensus.cc:2781] T f49fdae751cb4280a642fdaff98b52e8 P b1756f97ef304ef1ad449fbf0d69c3e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:59:25.532755 8423 raft_consensus.cc:482] T f49fdae751cb4280a642fdaff98b52e8 P b1756f97ef304ef1ad449fbf0d69c3e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:59:25.532780 8423 raft_consensus.cc:3037] T f49fdae751cb4280a642fdaff98b52e8 P b1756f97ef304ef1ad449fbf0d69c3e0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:59:25.533504 8423 raft_consensus.cc:504] T f49fdae751cb4280a642fdaff98b52e8 P b1756f97ef304ef1ad449fbf0d69c3e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a93275756742d3b9fa9c06d2bd8da4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34321 } } peers { permanent_uuid: "b1756f97ef304ef1ad449fbf0d69c3e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41169 } } peers { permanent_uuid: "737c56f6ae6643f69f44ff6f056b27a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37019 } }
I20260924 04:59:25.533627 8423 leader_election.cc:288] T f49fdae751cb4280a642fdaff98b52e8 P b1756f97ef304ef1ad449fbf0d69c3e0 [CANDIDATE]: Term 1 election: Requested vote from peers 00a93275756742d3b9fa9c06d2bd8da4 (127.0.98.1:34321), 737c56f6ae6643f69f44ff6f056b27a3 (127.0.98.3:37019)
I20260924 04:59:25.533794 8049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49fdae751cb4280a642fdaff98b52e8" candidate_uuid: "b1756f97ef304ef1ad449fbf0d69c3e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a93275756742d3b9fa9c06d2bd8da4"
I20260924 04:59:25.533776 8329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49fdae751cb4280a642fdaff98b52e8" candidate_uuid: "b1756f97ef304ef1ad449fbf0d69c3e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "737c56f6ae6643f69f44ff6f056b27a3"
I20260924 04:59:25.533864 8049 raft_consensus.cc:3037] T f49fdae751cb4280a642fdaff98b52e8 P 00a93275756742d3b9fa9c06d2bd8da4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:59:25.533869 8329 raft_consensus.cc:3037] T f49fdae751cb4280a642fdaff98b52e8 P 737c56f6ae6643f69f44ff6f056b27a3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:59:25.534416 8049 raft_consensus.cc:2454] T f49fdae751cb4280a642fdaff98b52e8 P 00a93275756742d3b9fa9c06d2bd8da4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1756f97ef304ef1ad449fbf0d69c3e0 in term 1.
I20260924 04:59:25.534569 8122 leader_election.cc:302] T f49fdae751cb4280a642fdaff98b52e8 P b1756f97ef304ef1ad449fbf0d69c3e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a93275756742d3b9fa9c06d2bd8da4, b1756f97ef304ef1ad449fbf0d69c3e0; no voters:
I20260924 04:59:25.534601 8329 raft_consensus.cc:2454] T f49fdae751cb4280a642fdaff98b52e8 P 737c56f6ae6643f69f44ff6f056b27a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1756f97ef304ef1ad449fbf0d69c3e0 in term 1.
I20260924 04:59:25.534682 8423 raft_consensus.cc:2781] T f49fdae751cb4280a642fdaff98b52e8 P b1756f97ef304ef1ad449fbf0d69c3e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:59:25.534803 8423 raft_consensus.cc:686] T f49fdae751cb4280a642fdaff98b52e8 P b1756f97ef304ef1ad449fbf0d69c3e0 [term 1 LEADER]: Becoming Leader. State: Replica: b1756f97ef304ef1ad449fbf0d69c3e0, State: Running, Role: LEADER
I20260924 04:59:25.534853 8423 consensus_queue.cc:237] T f49fdae751cb4280a642fdaff98b52e8 P b1756f97ef304ef1ad449fbf0d69c3e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a93275756742d3b9fa9c06d2bd8da4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34321 } } peers { permanent_uuid: "b1756f97ef304ef1ad449fbf0d69c3e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41169 } } peers { permanent_uuid: "737c56f6ae6643f69f44ff6f056b27a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37019 } }
I20260924 04:59:25.535504 7892 catalog_manager.cc:5495] T f49fdae751cb4280a642fdaff98b52e8 P b1756f97ef304ef1ad449fbf0d69c3e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to b1756f97ef304ef1ad449fbf0d69c3e0 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "b1756f97ef304ef1ad449fbf0d69c3e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00a93275756742d3b9fa9c06d2bd8da4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b1756f97ef304ef1ad449fbf0d69c3e0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41169 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "737c56f6ae6643f69f44ff6f056b27a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37019 } health_report { overall_health: UNKNOWN } } }
I20260924 04:59:25.565500 8402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:25.563475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48440 (local address 127.0.98.2:41169)
0924 04:59:25.563512 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:25.563513 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:25.563523 (+ 10us) server_negotiation.cc:407] Connection header received
0924 04:59:25.563552 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:25.563553 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:25.563563 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:25.563611 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:59:25.563692 (+ 81us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:25.564209 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:25.564959 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:25.565151 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:25.565215 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:25.565299 (+ 84us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:59:25.565323 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:59:25.565353 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:25.565384 (+ 31us) server_negotiation.cc:299] Negotiation successful
0924 04:59:25.565402 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260924 04:59:25.570144 8450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:25.567855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58609 (local address 127.0.98.3:37019)
0924 04:59:25.567928 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:25.567930 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:25.568024 (+ 94us) server_negotiation.cc:407] Connection header received
0924 04:59:25.568079 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:25.568080 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:25.568094 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:25.568139 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:59:25.568256 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:25.568705 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:25.569516 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:25.569708 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:25.569762 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:25.569914 (+ 152us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:59:25.569943 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:59:25.569983 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:25.570042 (+ 59us) server_negotiation.cc:299] Negotiation successful
0924 04:59:25.570065 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 04:59:25.570150 8502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:25.567780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37019 (local address 127.0.98.2:58609)
0924 04:59:25.567959 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:25.567970 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:25.568032 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:25.568154 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:25.568155 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:25.568157 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 04:59:25.568201 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:25.568202 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:25.568777 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:25.568778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:25.569445 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:25.569448 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:25.569772 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:25.569773 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:25.569827 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:25.569840 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 04:59:25.569993 (+ 153us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 04:59:25.569995 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 04:59:25.570030 (+ 35us) client_negotiation.cc:240] Negotiation successful
0924 04:59:25.570077 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":85,"threads_started":1}
I20260924 04:59:25.572095 8049 raft_consensus.cc:1261] T f49fdae751cb4280a642fdaff98b52e8 P 00a93275756742d3b9fa9c06d2bd8da4 [term 1 FOLLOWER]: Refusing update from remote peer b1756f97ef304ef1ad449fbf0d69c3e0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:59:25.572095 8329 raft_consensus.cc:1261] T f49fdae751cb4280a642fdaff98b52e8 P 737c56f6ae6643f69f44ff6f056b27a3 [term 1 FOLLOWER]: Refusing update from remote peer b1756f97ef304ef1ad449fbf0d69c3e0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:59:25.572304 8498 consensus_queue.cc:1035] T f49fdae751cb4280a642fdaff98b52e8 P b1756f97ef304ef1ad449fbf0d69c3e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a93275756742d3b9fa9c06d2bd8da4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:59:25.572414 8423 consensus_queue.cc:1035] T f49fdae751cb4280a642fdaff98b52e8 P b1756f97ef304ef1ad449fbf0d69c3e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "737c56f6ae6643f69f44ff6f056b27a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:59:25.578671 8402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:25.576507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56619 (local address 127.0.98.2:41169)
0924 04:59:25.576541 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:25.576543 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:25.576552 (+ 9us) server_negotiation.cc:407] Connection header received
0924 04:59:25.576570 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:25.576571 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:25.576579 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:25.576627 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:59:25.576725 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:25.577213 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:25.577995 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:25.578216 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:25.578272 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:25.578435 (+ 163us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:59:25.578470 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:59:25.578526 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:25.578580 (+ 54us) server_negotiation.cc:299] Negotiation successful
0924 04:59:25.578603 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260924 04:59:25.578687 8464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:25.576435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41169 (local address 127.0.98.3:56619)
0924 04:59:25.576474 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:25.576485 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:25.576522 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:25.576640 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:25.576641 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:25.576642 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 04:59:25.576680 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:25.576682 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:25.577295 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:25.577296 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:25.577924 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:25.577927 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:25.578303 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:25.578304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:25.578357 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:25.578363 (+ 6us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 04:59:25.578529 (+ 166us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 04:59:25.578531 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 04:59:25.578568 (+ 37us) client_negotiation.cc:240] Negotiation successful
0924 04:59:25.578613 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
I20260924 04:59:25.584702 7891 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49198:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:59:25.584798 7891 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49198:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:59:25.585364 7891 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8d8dc2f2261142e59c7c7ca2b072e3fd: Sending DeleteTablet for 3 replicas of tablet f49fdae751cb4280a642fdaff98b52e8
I20260924 04:59:25.585628 8169 tablet_service.cc:1514] Processing DeleteTablet for tablet f49fdae751cb4280a642fdaff98b52e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:59:25 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48420
I20260924 04:59:25.585634 8303 tablet_service.cc:1514] Processing DeleteTablet for tablet f49fdae751cb4280a642fdaff98b52e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:59:25 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33324
I20260924 04:59:25.585666 8029 tablet_service.cc:1514] Processing DeleteTablet for tablet f49fdae751cb4280a642fdaff98b52e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:59:25 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57468
I20260924 04:59:25.585726 8468 tablet_replica.cc:330] T f49fdae751cb4280a642fdaff98b52e8 P b1756f97ef304ef1ad449fbf0d69c3e0: stopping tablet replica
I20260924 04:59:25.585741 8470 tablet_replica.cc:330] T f49fdae751cb4280a642fdaff98b52e8 P 737c56f6ae6643f69f44ff6f056b27a3: stopping tablet replica
I20260924 04:59:25.585789 8470 raft_consensus.cc:2229] T f49fdae751cb4280a642fdaff98b52e8 P 737c56f6ae6643f69f44ff6f056b27a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:59:25.585778 8468 raft_consensus.cc:2229] T f49fdae751cb4280a642fdaff98b52e8 P b1756f97ef304ef1ad449fbf0d69c3e0 [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:59:25.585830 8470 raft_consensus.cc:2258] T f49fdae751cb4280a642fdaff98b52e8 P 737c56f6ae6643f69f44ff6f056b27a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:59:25.585852 8468 raft_consensus.cc:2258] T f49fdae751cb4280a642fdaff98b52e8 P b1756f97ef304ef1ad449fbf0d69c3e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:59:25.585927 8469 tablet_replica.cc:330] T f49fdae751cb4280a642fdaff98b52e8 P 00a93275756742d3b9fa9c06d2bd8da4: stopping tablet replica
I20260924 04:59:25.585978 8469 raft_consensus.cc:2229] T f49fdae751cb4280a642fdaff98b52e8 P 00a93275756742d3b9fa9c06d2bd8da4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:59:25.586004 8469 raft_consensus.cc:2258] T f49fdae751cb4280a642fdaff98b52e8 P 00a93275756742d3b9fa9c06d2bd8da4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:59:25.586100 8468 ts_tablet_manager.cc:1905] T f49fdae751cb4280a642fdaff98b52e8 P b1756f97ef304ef1ad449fbf0d69c3e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:59:25.586408 8469 ts_tablet_manager.cc:1905] T f49fdae751cb4280a642fdaff98b52e8 P 00a93275756742d3b9fa9c06d2bd8da4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:59:25.586436 8470 ts_tablet_manager.cc:1905] T f49fdae751cb4280a642fdaff98b52e8 P 737c56f6ae6643f69f44ff6f056b27a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:59:25.586467 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 7964
I20260924 04:59:25.587841 8470 ts_tablet_manager.cc:1918] T f49fdae751cb4280a642fdaff98b52e8 P 737c56f6ae6643f69f44ff6f056b27a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 04:59:25.587898 8470 log.cc:1196] T f49fdae751cb4280a642fdaff98b52e8 P 737c56f6ae6643f69f44ff6f056b27a3: Deleting WAL directory at /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225929848825-392-0/minicluster-data/ts-2/wal/wals/f49fdae751cb4280a642fdaff98b52e8
I20260924 04:59:25.588080 8470 ts_tablet_manager.cc:1939] T f49fdae751cb4280a642fdaff98b52e8 P 737c56f6ae6643f69f44ff6f056b27a3: Deleting consensus metadata
I20260924 04:59:25.588452 7876 catalog_manager.cc:4841] TS 737c56f6ae6643f69f44ff6f056b27a3 (127.0.98.3:37019): tablet f49fdae751cb4280a642fdaff98b52e8 (table test-table [id=33762fe4e582447fb8a68d6ee8da2863]) successfully deleted
I20260924 04:59:25.592188 8468 ts_tablet_manager.cc:1918] T f49fdae751cb4280a642fdaff98b52e8 P b1756f97ef304ef1ad449fbf0d69c3e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 04:59:25.592242 8468 log.cc:1196] T f49fdae751cb4280a642fdaff98b52e8 P b1756f97ef304ef1ad449fbf0d69c3e0: Deleting WAL directory at /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790225929848825-392-0/minicluster-data/ts-1/wal/wals/f49fdae751cb4280a642fdaff98b52e8
I20260924 04:59:25.592415 8468 ts_tablet_manager.cc:1939] T f49fdae751cb4280a642fdaff98b52e8 P b1756f97ef304ef1ad449fbf0d69c3e0: Deleting consensus metadata
I20260924 04:59:25.592679 7879 catalog_manager.cc:4841] TS b1756f97ef304ef1ad449fbf0d69c3e0 (127.0.98.2:41169): tablet f49fdae751cb4280a642fdaff98b52e8 (table test-table [id=33762fe4e582447fb8a68d6ee8da2863]) successfully deleted
W20260924 04:59:25.592764 7878 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34321 (error 108)
I20260924 04:59:25.593341 8507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:25.593130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34321 (local address 127.0.0.1:57484)
0924 04:59:25.593259 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:25.593304 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":74,"threads_started":1}
I20260924 04:59:25.593338 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 8104
W20260924 04:59:25.594038 7878 catalog_manager.cc:4584] TS 00a93275756742d3b9fa9c06d2bd8da4 (127.0.98.1:34321): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f49fdae751cb4280a642fdaff98b52e8: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34321: connect: Connection refused (error 111)
I20260924 04:59:25.598852 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 8244
I20260924 04:59:25.604022 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 7860
2026-09-24T04:59:25Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2887 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790225929848825-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:59:25 dist-test-slave-70st krb5kdc[8510](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:59:25 dist-test-slave-70st krb5kdc[8510](info): set up 2 sockets
Sep 24 04:59:25 dist-test-slave-70st krb5kdc[8510](info): commencing operation
krb5kdc: starting...
W20260924 04:59:26.553727 392 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790225929848825-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790225929848825-392-0/krb5kdc/test-user.keytab.
Sep 24 04:59:26 dist-test-slave-70st krb5kdc[8510](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225966, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:59:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:59:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:59:26.665016 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790225929848825-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790225929848825-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790225929848825-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42393
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42393
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:59:26.744084 8523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:59:26.744297 8523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:59:26.744325 8523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:59:26.745510 8523 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:59:26.745548 8523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:59:26.745566 8523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:59:26.745584 8523 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:59:26.745597 8523 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:59:26.747100 8523 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790225929848825-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790225929848825-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42393
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8523
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790225929848825-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 04:59:26.747340 8523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:59:26.747535 8523 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:59:26.752038 8531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:59:26.752045 8528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:59:26.752045 8529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:59:26.752346 8523 server_base.cc:1034] running on GCE node
I20260924 04:59:26.752570 8523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:59:26.752840 8523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:59:26.753984 8523 hybrid_clock.cc:648] HybridClock initialized: now 1790225966753967 us; error 34 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-24T04:59:26Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1159 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:59:26 dist-test-slave-70st krb5kdc[8536](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:59:26 dist-test-slave-70st krb5kdc[8536](info): set up 2 sockets
Sep 24 04:59:26 dist-test-slave-70st krb5kdc[8536](info): commencing operation
krb5kdc: starting...
W20260924 04:59:27.705652 392 mini_kdc.cc:121] Time spent starting KDC: real 0.923s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/krb5kdc/test-user.keytab.
Sep 24 04:59:27 dist-test-slave-70st krb5kdc[8536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225967, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:59:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:59:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:59:27.812110 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37517
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37517
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:59:27.890407 8549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:59:27.890595 8549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:59:27.890623 8549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:59:27.891734 8549 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:59:27.891772 8549 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:59:27.891791 8549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:59:27.891806 8549 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:59:27.891821 8549 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:59:27.893280 8549 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37517
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37517
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8549
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 04:59:27.893518 8549 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:59:27.893710 8549 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:59:27.898090 8557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:59:27.898159 8549 server_base.cc:1034] running on GCE node
W20260924 04:59:27.898098 8554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:59:27.898101 8555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:59:27.898541 8549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:59:27.898830 8549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:59:27.899971 8549 hybrid_clock.cc:648] HybridClock initialized: now 1790225967899955 us; error 32 us; skew 500 ppm
Sep 24 04:59:27 dist-test-slave-70st krb5kdc[8536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225967, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:59:27.902262 8549 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:59:27.902637 8549 webserver.cc:460] Webserver started at http://127.0.98.62:36069/ using document root <none> and password file <none>
I20260924 04:59:27.902848 8549 fs_manager.cc:359] Metadata directory not provided
I20260924 04:59:27.902892 8549 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:59:27.903007 8549 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:59:27.903972 8549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/master-0/data/instance:
uuid: "4bd486bddca4465dbbf17774f9589eb3"
format_stamp: "Formatted at 2026-09-24 04:59:27 on dist-test-slave-70st"
server_key: "e88702a76a19871a70e8e40f8065d836"
server_key_iv: "570d27be8edac4c704f18ace2fe5baf9"
server_key_version: "encryptionkey@0"
I20260924 04:59:27.904277 8549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/master-0/wal/instance:
uuid: "4bd486bddca4465dbbf17774f9589eb3"
format_stamp: "Formatted at 2026-09-24 04:59:27 on dist-test-slave-70st"
server_key: "e88702a76a19871a70e8e40f8065d836"
server_key_iv: "570d27be8edac4c704f18ace2fe5baf9"
server_key_version: "encryptionkey@0"
I20260924 04:59:27.905409 8549 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 04:59:27.906111 8564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:27.906255 8549 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 04:59:27.906332 8549 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/master-0/wal
uuid: "4bd486bddca4465dbbf17774f9589eb3"
format_stamp: "Formatted at 2026-09-24 04:59:27 on dist-test-slave-70st"
server_key: "e88702a76a19871a70e8e40f8065d836"
server_key_iv: "570d27be8edac4c704f18ace2fe5baf9"
server_key_version: "encryptionkey@0"
I20260924 04:59:27.906392 8549 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:59:27.918717 8549 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:59:27.920660 8549 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:59:27.920790 8549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:59:27.924319 8549 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37517
I20260924 04:59:27.924357 8616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37517 every 8 connection(s)
I20260924 04:59:27.924796 8549 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:59:27.925357 8617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:59:27.925879 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 8549
I20260924 04:59:27.925952 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/master-0/wal/instance
I20260924 04:59:27.926098 392 external_mini_cluster.cc:1428] Setting key c2ad288d4033ad305ac2ce25aa4ff21c
I20260924 04:59:27.927901 8617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bd486bddca4465dbbf17774f9589eb3: Bootstrap starting.
I20260924 04:59:27.928455 8617 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4bd486bddca4465dbbf17774f9589eb3: Neither blocks nor log segments found. Creating new log.
I20260924 04:59:27.928690 8617 log.cc:824] T 00000000000000000000000000000000 P 4bd486bddca4465dbbf17774f9589eb3: Log is configured to *not* fsync() on all Append() calls
I20260924 04:59:27.929284 8617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bd486bddca4465dbbf17774f9589eb3: No bootstrap required, opened a new log
I20260924 04:59:27.931393 8617 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4bd486bddca4465dbbf17774f9589eb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bd486bddca4465dbbf17774f9589eb3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37517 } }
I20260924 04:59:27.931517 8617 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4bd486bddca4465dbbf17774f9589eb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:59:27.931550 8617 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4bd486bddca4465dbbf17774f9589eb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bd486bddca4465dbbf17774f9589eb3, State: Initialized, Role: FOLLOWER
I20260924 04:59:27.931670 8617 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4bd486bddca4465dbbf17774f9589eb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bd486bddca4465dbbf17774f9589eb3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37517 } }
I20260924 04:59:27.931738 8617 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4bd486bddca4465dbbf17774f9589eb3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:59:27.931782 8617 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4bd486bddca4465dbbf17774f9589eb3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:59:27.931828 8617 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4bd486bddca4465dbbf17774f9589eb3 [term 0 FOLLOWER]: Advancing to term 1
Sep 24 04:59:27 dist-test-slave-70st krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225967, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:59:27.932312 8617 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4bd486bddca4465dbbf17774f9589eb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bd486bddca4465dbbf17774f9589eb3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37517 } }
I20260924 04:59:27.932400 8617 leader_election.cc:302] T 00000000000000000000000000000000 P 4bd486bddca4465dbbf17774f9589eb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4bd486bddca4465dbbf17774f9589eb3; no voters:
I20260924 04:59:27.932552 8617 leader_election.cc:288] T 00000000000000000000000000000000 P 4bd486bddca4465dbbf17774f9589eb3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:59:27.932602 8622 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4bd486bddca4465dbbf17774f9589eb3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:59:27.932739 8622 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4bd486bddca4465dbbf17774f9589eb3 [term 1 LEADER]: Becoming Leader. State: Replica: 4bd486bddca4465dbbf17774f9589eb3, State: Running, Role: LEADER
I20260924 04:59:27.932788 8617 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4bd486bddca4465dbbf17774f9589eb3 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:59:27.932817 8622 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4bd486bddca4465dbbf17774f9589eb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bd486bddca4465dbbf17774f9589eb3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37517 } }
I20260924 04:59:27.933264 8623 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bd486bddca4465dbbf17774f9589eb3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4bd486bddca4465dbbf17774f9589eb3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bd486bddca4465dbbf17774f9589eb3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37517 } } }
I20260924 04:59:27.933336 8623 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bd486bddca4465dbbf17774f9589eb3 [sys.catalog]: This master's current role is: LEADER
I20260924 04:59:27.933470 8624 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bd486bddca4465dbbf17774f9589eb3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4bd486bddca4465dbbf17774f9589eb3. Latest consensus state: current_term: 1 leader_uuid: "4bd486bddca4465dbbf17774f9589eb3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bd486bddca4465dbbf17774f9589eb3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37517 } } }
I20260924 04:59:27.933532 8624 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bd486bddca4465dbbf17774f9589eb3 [sys.catalog]: This master's current role is: LEADER
W20260924 04:59:27.934725 8638 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4bd486bddca4465dbbf17774f9589eb3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:59:27.934783 8638 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:59:27.934867 8631 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:59:27.935060 8631 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:59:27.936074 8620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:27.926552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47510 (local address 127.0.98.62:37517)
0924 04:59:27.926681 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:27.926689 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:27.926712 (+ 23us) server_negotiation.cc:407] Connection header received
0924 04:59:27.927094 (+ 382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:27.927100 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:27.927141 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:27.927249 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:27.927972 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:27.928874 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:27.929685 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:27.930312 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:27.930423 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:27.932449 (+ 2026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:27.932453 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:27.932453 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:59:27.932464 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:27.934350 (+ 1886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:27.934691 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:27.934694 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:27.934694 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:27.934790 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:27.935019 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:27.935020 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:27.935020 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:27.935307 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:27.935433 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:27.935795 (+ 362us) server_negotiation.cc:299] Negotiation successful
0924 04:59:27.935896 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":51,"threads_started":1}
I20260924 04:59:27.936436 8631 catalog_manager.cc:1347] Generated new cluster ID: 076b9d712fb74b86a5542a43eb5c2901
I20260924 04:59:27.936488 8631 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:59:27.946628 8631 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:59:27.947124 8631 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:59:27.955682 8631 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4bd486bddca4465dbbf17774f9589eb3: Generated new TSK 0
I20260924 04:59:27.955837 8631 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:59:27.987677 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37517
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:59:28.083002 8643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:59:28.083210 8643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:59:28.083241 8643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:59:28.084367 8643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:59:28.084408 8643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:59:28.084444 8643 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:59:28.085948 8643 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8643
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 04:59:28.086184 8643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:59:28.086387 8643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:59:28.090967 8649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:59:28.091007 8651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:59:28.090996 8648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:59:28.091189 8643 server_base.cc:1034] running on GCE node
I20260924 04:59:28.091395 8643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:59:28.091602 8643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:59:28.092742 8643 hybrid_clock.cc:648] HybridClock initialized: now 1790225968092718 us; error 38 us; skew 500 ppm
Sep 24 04:59:28 dist-test-slave-70st krb5kdc[8536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225968, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:59:28.095110 8643 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:59:28.095543 8643 webserver.cc:460] Webserver started at http://127.0.98.1:41347/ using document root <none> and password file <none>
I20260924 04:59:28.095741 8643 fs_manager.cc:359] Metadata directory not provided
I20260924 04:59:28.095784 8643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:59:28.095912 8643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:59:28.096925 8643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/ts-0/data/instance:
uuid: "52e7210eb3b04cb5ad61d844ae90a063"
format_stamp: "Formatted at 2026-09-24 04:59:28 on dist-test-slave-70st"
server_key: "8abd237f95808bec6f61defb97cb1a7a"
server_key_iv: "4433b342cf002849b8aed8cd93a5f5f5"
server_key_version: "encryptionkey@0"
I20260924 04:59:28.097234 8643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/ts-0/wal/instance:
uuid: "52e7210eb3b04cb5ad61d844ae90a063"
format_stamp: "Formatted at 2026-09-24 04:59:28 on dist-test-slave-70st"
server_key: "8abd237f95808bec6f61defb97cb1a7a"
server_key_iv: "4433b342cf002849b8aed8cd93a5f5f5"
server_key_version: "encryptionkey@0"
I20260924 04:59:28.098489 8643 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 04:59:28.099128 8658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:28.099325 8643 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 04:59:28.099403 8643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/ts-0/wal
uuid: "52e7210eb3b04cb5ad61d844ae90a063"
format_stamp: "Formatted at 2026-09-24 04:59:28 on dist-test-slave-70st"
server_key: "8abd237f95808bec6f61defb97cb1a7a"
server_key_iv: "4433b342cf002849b8aed8cd93a5f5f5"
server_key_version: "encryptionkey@0"
I20260924 04:59:28.099457 8643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:59:28.122525 8643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:59:28.124869 8643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:59:28.125010 8643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:59:28.125223 8643 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:59:28.125537 8643 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:59:28.125572 8643 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:28.125602 8643 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:59:28.125625 8643 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:28.130930 8643 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46125
I20260924 04:59:28.130970 8771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46125 every 8 connection(s)
I20260924 04:59:28.131268 8643 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:59:28.132839 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 8643
I20260924 04:59:28.132964 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:59:28.133173 392 external_mini_cluster.cc:1428] Setting key a0970955bfaaa1c6454bf4d1bde13050
Sep 24 04:59:28 dist-test-slave-70st krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225968, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 04:59:28.140952 8620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:28.132000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57377 (local address 127.0.98.62:37517)
0924 04:59:28.132067 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:28.132069 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:28.132233 (+ 164us) server_negotiation.cc:407] Connection header received
0924 04:59:28.133402 (+ 1169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:28.133404 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:28.133429 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:28.133494 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:28.134291 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:28.134825 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:28.135862 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:28.136157 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:28.136220 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:28.138224 (+ 2004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:28.138226 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:28.138226 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:59:28.138234 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:28.139711 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:28.140077 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:28.140079 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:28.140079 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:28.140134 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:28.140334 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:28.140335 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:28.140335 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:28.140467 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:28.140529 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:28.140760 (+ 231us) server_negotiation.cc:299] Negotiation successful
0924 04:59:28.140858 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 04:59:28.141216 8774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:28.131964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.1:57377)
0924 04:59:28.132112 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:28.132153 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:28.133301 (+ 1148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:28.133570 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:28.133574 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:28.133904 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:28.134200 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:28.134207 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:28.134915 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:28.134917 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:28.135782 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:28.135784 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:28.136229 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:28.136230 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:28.136323 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:28.136792 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:28.136801 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:28.138138 (+ 1337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:59:28.139790 (+ 1652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:28.139796 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:28.139797 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:28.140008 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:28.140195 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:28.140195 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:28.140195 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:28.140280 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:28.140525 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:59:28.140528 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:59:28.140636 (+ 108us) client_negotiation.cc:769] Sending connection context
0924 04:59:28.140720 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 04:59:28.140865 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":67,"threads_started":1}
I20260924 04:59:28.141743 8772 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37517
I20260924 04:59:28.141839 8772 heartbeater.cc:464] Registering TS with master...
I20260924 04:59:28.142004 8772 heartbeater.cc:511] Master 127.0.98.62:37517 requested a full tablet report, sending...
I20260924 04:59:28.142513 8581 ts_manager.cc:194] Registered new tserver with Master: 52e7210eb3b04cb5ad61d844ae90a063 (127.0.98.1:46125)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 04:59:28.143185 8581 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:57377
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:59:28.156268 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37517
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:59:28.233433 8777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:59:28.233644 8777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:59:28.233666 8777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:59:28.234836 8777 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:59:28.234876 8777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:59:28.234901 8777 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:59:28.236353 8777 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8777
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 04:59:28.236565 8777 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:59:28.236780 8777 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:59:28.241562 8785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:59:28.241600 8783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:59:28.241650 8777 server_base.cc:1034] running on GCE node
W20260924 04:59:28.241562 8782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:59:28.241976 8777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:59:28.242166 8777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:59:28.243314 8777 hybrid_clock.cc:648] HybridClock initialized: now 1790225968243304 us; error 31 us; skew 500 ppm
Sep 24 04:59:28 dist-test-slave-70st krb5kdc[8536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225968, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:59:28.245592 8777 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:59:28.246025 8777 webserver.cc:460] Webserver started at http://127.0.98.2:34853/ using document root <none> and password file <none>
I20260924 04:59:28.246213 8777 fs_manager.cc:359] Metadata directory not provided
I20260924 04:59:28.246268 8777 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:59:28.246393 8777 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:59:28.247416 8777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/ts-1/data/instance:
uuid: "330ef213c92640339e66f9d2dce79b24"
format_stamp: "Formatted at 2026-09-24 04:59:28 on dist-test-slave-70st"
server_key: "b3bd291ae8079c218e04324335f0980a"
server_key_iv: "20a05c14c9c2c33b218c99e730c044cf"
server_key_version: "encryptionkey@0"
I20260924 04:59:28.247735 8777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/ts-1/wal/instance:
uuid: "330ef213c92640339e66f9d2dce79b24"
format_stamp: "Formatted at 2026-09-24 04:59:28 on dist-test-slave-70st"
server_key: "b3bd291ae8079c218e04324335f0980a"
server_key_iv: "20a05c14c9c2c33b218c99e730c044cf"
server_key_version: "encryptionkey@0"
I20260924 04:59:28.248919 8777 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 04:59:28.249598 8792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:28.249743 8777 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 04:59:28.249821 8777 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/ts-1/wal
uuid: "330ef213c92640339e66f9d2dce79b24"
format_stamp: "Formatted at 2026-09-24 04:59:28 on dist-test-slave-70st"
server_key: "b3bd291ae8079c218e04324335f0980a"
server_key_iv: "20a05c14c9c2c33b218c99e730c044cf"
server_key_version: "encryptionkey@0"
I20260924 04:59:28.249887 8777 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:59:28.257302 8777 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:59:28.258885 8777 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:59:28.259003 8777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:59:28.259182 8777 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:59:28.259457 8777 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:59:28.259488 8777 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:28.259516 8777 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:59:28.259539 8777 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:28.264756 8777 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36057
I20260924 04:59:28.264811 8905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36057 every 8 connection(s)
I20260924 04:59:28.265084 8777 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:59:28.270355 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 8777
I20260924 04:59:28.270449 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:59:28.270607 392 external_mini_cluster.cc:1428] Setting key 99970330c22db60ba42e18691fdab220
Sep 24 04:59:28 dist-test-slave-70st krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225968, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:59:28.274209 8620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:28.265755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45127 (local address 127.0.98.62:37517)
0924 04:59:28.265804 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:28.265806 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:28.266143 (+ 337us) server_negotiation.cc:407] Connection header received
0924 04:59:28.267157 (+ 1014us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:28.267158 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:28.267168 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:28.267233 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:28.267835 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:28.268375 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:28.269304 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:28.269554 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:28.269607 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:28.271620 (+ 2013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:28.271621 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:28.271621 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:59:28.271629 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:28.272966 (+ 1337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:28.273356 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:28.273357 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:28.273357 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:28.273411 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:28.273668 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:28.273669 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:28.273669 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:28.273763 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:28.273808 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:28.274085 (+ 277us) server_negotiation.cc:299] Negotiation successful
0924 04:59:28.274144 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 04:59:28.274444 8908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:28.265745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.2:45127)
0924 04:59:28.266020 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:28.266066 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:28.267048 (+ 982us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:28.267297 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:28.267301 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:28.267545 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:28.267733 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:28.267737 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:28.268446 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:28.268446 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:28.269227 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:28.269228 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:28.269625 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:28.269625 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:28.269726 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:28.270136 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:28.270143 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:28.271537 (+ 1394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:59:28.273045 (+ 1508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:28.273051 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:28.273053 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:28.273277 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:28.273485 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:28.273485 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:28.273486 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:28.273613 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:28.273813 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:59:28.273817 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:59:28.273969 (+ 152us) client_negotiation.cc:769] Sending connection context
0924 04:59:28.274073 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 04:59:28.274225 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":79,"threads_started":1}
I20260924 04:59:28.274895 8906 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37517
I20260924 04:59:28.275004 8906 heartbeater.cc:464] Registering TS with master...
I20260924 04:59:28.275169 8906 heartbeater.cc:511] Master 127.0.98.62:37517 requested a full tablet report, sending...
I20260924 04:59:28.275528 8581 ts_manager.cc:194] Registered new tserver with Master: 330ef213c92640339e66f9d2dce79b24 (127.0.98.2:36057)
I20260924 04:59:28.276028 8581 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:45127
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:59:28.294095 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37517
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:59:28.371698 8911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:59:28.371915 8911 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:59:28.371945 8911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:59:28.373085 8911 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:59:28.373123 8911 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:59:28.373157 8911 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:59:28.374665 8911 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8911
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 04:59:28.374902 8911 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:59:28.375123 8911 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:59:28.379807 8916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:59:28.379815 8917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:59:28.379973 8911 server_base.cc:1034] running on GCE node
W20260924 04:59:28.379814 8919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:59:28.380256 8911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:59:28.380477 8911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:59:28.381616 8911 hybrid_clock.cc:648] HybridClock initialized: now 1790225968381593 us; error 31 us; skew 500 ppm
Sep 24 04:59:28 dist-test-slave-70st krb5kdc[8536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225968, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:59:28.384078 8911 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:59:28.384524 8911 webserver.cc:460] Webserver started at http://127.0.98.3:42627/ using document root <none> and password file <none>
I20260924 04:59:28.384739 8911 fs_manager.cc:359] Metadata directory not provided
I20260924 04:59:28.384788 8911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:59:28.384905 8911 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:59:28.385954 8911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/ts-2/data/instance:
uuid: "e4b5732ab3c6431bb161ee9a4aff6bda"
format_stamp: "Formatted at 2026-09-24 04:59:28 on dist-test-slave-70st"
server_key: "10abcecf4c39c297f1f816e44042e237"
server_key_iv: "8bc7aabf53250f14d5e8698a7c0586b7"
server_key_version: "encryptionkey@0"
I20260924 04:59:28.386286 8911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e4b5732ab3c6431bb161ee9a4aff6bda"
format_stamp: "Formatted at 2026-09-24 04:59:28 on dist-test-slave-70st"
server_key: "10abcecf4c39c297f1f816e44042e237"
server_key_iv: "8bc7aabf53250f14d5e8698a7c0586b7"
server_key_version: "encryptionkey@0"
I20260924 04:59:28.387531 8911 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 04:59:28.388393 8926 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:28.388546 8911 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 04:59:28.388628 8911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/ts-2/wal
uuid: "e4b5732ab3c6431bb161ee9a4aff6bda"
format_stamp: "Formatted at 2026-09-24 04:59:28 on dist-test-slave-70st"
server_key: "10abcecf4c39c297f1f816e44042e237"
server_key_iv: "8bc7aabf53250f14d5e8698a7c0586b7"
server_key_version: "encryptionkey@0"
I20260924 04:59:28.388690 8911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:59:28.407094 8911 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:59:28.409260 8911 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:59:28.409405 8911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:59:28.409662 8911 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:59:28.409957 8911 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:59:28.409991 8911 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:28.410018 8911 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:59:28.410043 8911 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:28.415328 8911 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39663
I20260924 04:59:28.415370 9039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39663 every 8 connection(s)
I20260924 04:59:28.415709 8911 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:59:28.418277 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 8911
I20260924 04:59:28.418385 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:59:28.418536 392 external_mini_cluster.cc:1428] Setting key 3a81e4e56613e8bddbd23cce6a68c81d
Sep 24 04:59:28 dist-test-slave-70st krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225968, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:59:28.424834 8620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:28.416445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47885 (local address 127.0.98.62:37517)
0924 04:59:28.416510 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:28.416513 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:28.416752 (+ 239us) server_negotiation.cc:407] Connection header received
0924 04:59:28.417683 (+ 931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:28.417684 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:28.417695 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:28.417749 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:28.418391 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:28.419067 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:28.420218 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:28.420402 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:28.420446 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:28.422512 (+ 2066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:28.422514 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:28.422514 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:59:28.422520 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:28.423821 (+ 1301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:28.424160 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:28.424161 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:28.424161 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:28.424205 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:28.424385 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:28.424386 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:28.424386 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:28.424475 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:28.424517 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:28.424718 (+ 201us) server_negotiation.cc:299] Negotiation successful
0924 04:59:28.424768 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 04:59:28.425041 9042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:28.416442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.3:47885)
0924 04:59:28.416617 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:28.416664 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:28.417592 (+ 928us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:28.417814 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:28.417818 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:28.418106 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:28.418300 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:28.418304 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:28.419252 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:28.419253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:28.420145 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:28.420147 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:28.420455 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:28.420455 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:28.420552 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:28.421016 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:28.421026 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:28.422430 (+ 1404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:59:28.423894 (+ 1464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:28.423900 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:28.423901 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:28.424104 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:28.424249 (+ 145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:28.424249 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:28.424249 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:28.424343 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:28.424518 (+ 175us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:59:28.424521 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:59:28.424628 (+ 107us) client_negotiation.cc:769] Sending connection context
0924 04:59:28.424702 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 04:59:28.424818 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":96,"threads_started":1}
I20260924 04:59:28.425535 9040 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37517
I20260924 04:59:28.425635 9040 heartbeater.cc:464] Registering TS with master...
I20260924 04:59:28.425792 9040 heartbeater.cc:511] Master 127.0.98.62:37517 requested a full tablet report, sending...
I20260924 04:59:28.426110 8581 ts_manager.cc:194] Registered new tserver with Master: e4b5732ab3c6431bb161ee9a4aff6bda (127.0.98.3:39663)
I20260924 04:59:28.426445 8581 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47885
I20260924 04:59:28.430081 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260924 04:59:28.494108 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 8643
I20260924 04:59:28.497853 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 8777
I20260924 04:59:28.501638 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 8911
I20260924 04:59:28.505162 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 8549
I20260924 04:59:28.509300 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37517
--webserver_interface=127.0.98.62
--webserver_port=36069
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37517
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:59:28.586970 9049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:59:28.587178 9049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:59:28.587208 9049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:59:28.588348 9049 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:59:28.588388 9049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:59:28.588407 9049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:59:28.588423 9049 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:59:28.588438 9049 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:59:28.589920 9049 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37517
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37517
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=36069
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9049
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 04:59:28.590157 9049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:59:28.590349 9049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:59:28.595024 9054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:59:28.595039 9057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:59:28.595177 9049 server_base.cc:1034] running on GCE node
W20260924 04:59:28.595050 9055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:59:28.595487 9049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:59:28.595795 9049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:59:28.596940 9049 hybrid_clock.cc:648] HybridClock initialized: now 1790225968596922 us; error 29 us; skew 500 ppm
Sep 24 04:59:28 dist-test-slave-70st krb5kdc[8536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225968, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:59:28.599396 9049 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:59:28.599812 9049 webserver.cc:460] Webserver started at http://127.0.98.62:36069/ using document root <none> and password file <none>
I20260924 04:59:28.600004 9049 fs_manager.cc:359] Metadata directory not provided
I20260924 04:59:28.600067 9049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:59:28.601352 9049 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 04:59:28.601974 9064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:28.602135 9049 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:28.602221 9049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/master-0/wal
uuid: "4bd486bddca4465dbbf17774f9589eb3"
format_stamp: "Formatted at 2026-09-24 04:59:27 on dist-test-slave-70st"
server_key: "e88702a76a19871a70e8e40f8065d836"
server_key_iv: "570d27be8edac4c704f18ace2fe5baf9"
server_key_version: "encryptionkey@0"
I20260924 04:59:28.602551 9049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:59:28.609632 9049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:59:28.611743 9049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:59:28.611882 9049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:59:28.615895 9049 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37517
I20260924 04:59:28.615934 9116 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37517 every 8 connection(s)
I20260924 04:59:28.616427 9049 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:59:28.617734 9117 sys_catalog.cc:263] Verifying existing consensus state
I20260924 04:59:28.618357 9117 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bd486bddca4465dbbf17774f9589eb3: Bootstrap starting.
I20260924 04:59:28.623311 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 9049
I20260924 04:59:28.623566 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:46125
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:37517
--webserver_port=41347
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 04:59:28.626444 9117 log.cc:824] T 00000000000000000000000000000000 P 4bd486bddca4465dbbf17774f9589eb3: Log is configured to *not* fsync() on all Append() calls
I20260924 04:59:28.628028 9117 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bd486bddca4465dbbf17774f9589eb3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 04:59:28.628278 9117 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bd486bddca4465dbbf17774f9589eb3: Bootstrap complete.
I20260924 04:59:28.630347 9117 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4bd486bddca4465dbbf17774f9589eb3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bd486bddca4465dbbf17774f9589eb3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37517 } }
I20260924 04:59:28.630563 9117 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4bd486bddca4465dbbf17774f9589eb3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bd486bddca4465dbbf17774f9589eb3, State: Initialized, Role: FOLLOWER
I20260924 04:59:28.630707 9117 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4bd486bddca4465dbbf17774f9589eb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bd486bddca4465dbbf17774f9589eb3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37517 } }
I20260924 04:59:28.630781 9117 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4bd486bddca4465dbbf17774f9589eb3 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:59:28.630831 9117 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4bd486bddca4465dbbf17774f9589eb3 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:59:28.630878 9117 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4bd486bddca4465dbbf17774f9589eb3 [term 1 FOLLOWER]: Advancing to term 2
I20260924 04:59:28.631821 9117 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4bd486bddca4465dbbf17774f9589eb3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bd486bddca4465dbbf17774f9589eb3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37517 } }
I20260924 04:59:28.631914 9117 leader_election.cc:302] T 00000000000000000000000000000000 P 4bd486bddca4465dbbf17774f9589eb3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4bd486bddca4465dbbf17774f9589eb3; no voters:
I20260924 04:59:28.632064 9117 leader_election.cc:288] T 00000000000000000000000000000000 P 4bd486bddca4465dbbf17774f9589eb3 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 04:59:28.632128 9121 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4bd486bddca4465dbbf17774f9589eb3 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 04:59:28.632318 9117 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4bd486bddca4465dbbf17774f9589eb3 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:59:28.632328 9121 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4bd486bddca4465dbbf17774f9589eb3 [term 2 LEADER]: Becoming Leader. State: Replica: 4bd486bddca4465dbbf17774f9589eb3, State: Running, Role: LEADER
I20260924 04:59:28.632454 9121 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4bd486bddca4465dbbf17774f9589eb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bd486bddca4465dbbf17774f9589eb3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37517 } }
I20260924 04:59:28.632846 9123 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bd486bddca4465dbbf17774f9589eb3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4bd486bddca4465dbbf17774f9589eb3. Latest consensus state: current_term: 2 leader_uuid: "4bd486bddca4465dbbf17774f9589eb3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bd486bddca4465dbbf17774f9589eb3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37517 } } }
I20260924 04:59:28.632984 9123 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bd486bddca4465dbbf17774f9589eb3 [sys.catalog]: This master's current role is: LEADER
I20260924 04:59:28.633244 9122 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bd486bddca4465dbbf17774f9589eb3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "4bd486bddca4465dbbf17774f9589eb3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bd486bddca4465dbbf17774f9589eb3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37517 } } }
I20260924 04:59:28.633292 9132 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:59:28.633852 9132 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:59:28.633960 9132 catalog_manager.cc:1259] Loaded cluster ID: 076b9d712fb74b86a5542a43eb5c2901
I20260924 04:59:28.633983 9132 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:59:28.635103 9132 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:59:28.635263 9132 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4bd486bddca4465dbbf17774f9589eb3: Loaded TSK: 0
I20260924 04:59:28.635463 9132 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:59:28.635607 9122 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bd486bddca4465dbbf17774f9589eb3 [sys.catalog]: This master's current role is: LEADER
W20260924 04:59:28.702997 9119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:59:28.703208 9119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:59:28.703239 9119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:59:28.704399 9119 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:59:28.704438 9119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:59:28.704464 9119 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:59:28.705971 9119 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:46125
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=41347
--tserver_master_addrs=127.0.98.62:37517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9119
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 04:59:28.706212 9119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:59:28.706425 9119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:59:28.711117 9144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:59:28.711119 9143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:59:28.711158 9146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:59:28.711552 9119 server_base.cc:1034] running on GCE node
I20260924 04:59:28.711746 9119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:59:28.711963 9119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:59:28.713121 9119 hybrid_clock.cc:648] HybridClock initialized: now 1790225968713111 us; error 28 us; skew 500 ppm
Sep 24 04:59:28 dist-test-slave-70st krb5kdc[8536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225968, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:59:28.715777 9119 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260924 04:59:28.716231 9119 webserver.cc:460] Webserver started at http://127.0.98.1:41347/ using document root <none> and password file <none>
I20260924 04:59:28.716445 9119 fs_manager.cc:359] Metadata directory not provided
I20260924 04:59:28.716499 9119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:59:28.717809 9119 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 04:59:28.718354 9153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:28.718506 9119 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 04:59:28.718575 9119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/ts-0/wal
uuid: "52e7210eb3b04cb5ad61d844ae90a063"
format_stamp: "Formatted at 2026-09-24 04:59:28 on dist-test-slave-70st"
server_key: "8abd237f95808bec6f61defb97cb1a7a"
server_key_iv: "4433b342cf002849b8aed8cd93a5f5f5"
server_key_version: "encryptionkey@0"
I20260924 04:59:28.718915 9119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:59:28.727530 9119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:59:28.729596 9119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:59:28.729712 9119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:59:28.729923 9119 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:59:28.730242 9119 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:59:28.730278 9119 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:28.730298 9119 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:59:28.730310 9119 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:28.737257 9119 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46125
I20260924 04:59:28.737383 9266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46125 every 8 connection(s)
I20260924 04:59:28.737802 9119 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:59:28.738574 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 9119
I20260924 04:59:28.738850 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:36057
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:37517
--webserver_port=34853
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 04:59:28 dist-test-slave-70st krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:59:28 dist-test-slave-70st krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:59:28.750473 9270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:28.740406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.1:38637)
0924 04:59:28.740587 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:28.740635 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:28.741241 (+ 606us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:28.741983 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:28.741988 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:28.742649 (+ 661us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:28.742929 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:28.742935 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:28.744557 (+ 1622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:28.744558 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:28.745846 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:28.745849 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:28.746834 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:28.746835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:28.746984 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:28.747581 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:28.747591 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:28.750275 (+ 2684us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":89,"threads_started":1}
W20260924 04:59:28.750660 9267 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37517 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:37517: Client connection negotiation failed: client connection to 127.0.98.62:37517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
W20260924 04:59:28.750855 9271 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:59:28.740672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38637 (local address 127.0.98.62:37517)
0924 04:59:28.741530 (+ 858us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:28.741540 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:28.741582 (+ 42us) server_negotiation.cc:407] Connection header received
0924 04:59:28.741713 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:28.741719 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:28.741781 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:28.741911 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:28.743050 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:28.744450 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:28.745927 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:28.746616 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:28.746719 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:28.750421 (+ 3702us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:38637: BlockingRecv error: recv got EOF from 127.0.98.1:38637 (error 108)
Metrics: {"server-negotiator.queue_time_us":806,"thread_start_us":92,"threads_started":1}
Sep 24 04:59:28 dist-test-slave-70st krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:59:28 dist-test-slave-70st krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:59:28.756093 9270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:28.750867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.1:60143)
0924 04:59:28.751206 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:28.751216 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:28.751273 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:28.751395 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:28.751395 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:28.751529 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:28.751578 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:28.751579 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:28.752444 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:28.752444 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:28.753274 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:28.753276 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:28.753712 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:28.753713 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:28.753775 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:28.754177 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:28.754182 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:28.756027 (+ 1845us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":324}
W20260924 04:59:28.756312 9271 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:59:28.750955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60143 (local address 127.0.98.62:37517)
0924 04:59:28.750987 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:28.750989 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:28.751272 (+ 283us) server_negotiation.cc:407] Connection header received
0924 04:59:28.751318 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:28.751319 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:28.751330 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:28.751387 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:28.751633 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:28.752376 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:28.753339 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:28.753647 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:28.753699 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:28.756119 (+ 2420us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:60143: BlockingRecv error: recv got EOF from 127.0.98.1:60143 (error 108)
Metrics: {"server-negotiator.queue_time_us":16,"thread_start_us":75,"threads_started":1}
Sep 24 04:59:28 dist-test-slave-70st krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:59:28 dist-test-slave-70st krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:59:28.760591 9270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:28.756314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.1:45493)
0924 04:59:28.756419 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:28.756427 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:28.756458 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:28.756611 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:28.756612 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:28.756740 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:28.756771 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:28.756772 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:28.757591 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:28.757592 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:28.758380 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:28.758382 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:28.758680 (+ 298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:28.758681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:28.758729 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:28.759054 (+ 325us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:28.759058 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:28.760550 (+ 1492us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":92}
W20260924 04:59:28.760701 9267 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 04:59:28.760738 9271 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:59:28.756389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45493 (local address 127.0.98.62:37517)
0924 04:59:28.756489 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:28.756491 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:28.756499 (+ 8us) server_negotiation.cc:407] Connection header received
0924 04:59:28.756537 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:28.756538 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:28.756547 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:28.756598 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:28.756830 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:28.757525 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:28.758453 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:28.758620 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:28.758670 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:28.760685 (+ 2015us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:45493: BlockingRecv error: recv got EOF from 127.0.98.1:45493 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
W20260924 04:59:28.822172 9269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:59:28.822386 9269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:59:28.822409 9269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:59:28.823560 9269 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:59:28.823598 9269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:59:28.823623 9269 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:59:28.824994 9269 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:36057
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=34853
--tserver_master_addrs=127.0.98.62:37517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9269
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 04:59:28.825198 9269 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:59:28.825402 9269 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:59:28.830049 9277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:59:28.830044 9278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:59:28.830168 9280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:59:28.830183 9269 server_base.cc:1034] running on GCE node
I20260924 04:59:28.830444 9269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:59:28.830652 9269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:59:28.831802 9269 hybrid_clock.cc:648] HybridClock initialized: now 1790225968831791 us; error 30 us; skew 500 ppm
Sep 24 04:59:28 dist-test-slave-70st krb5kdc[8536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225968, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:59:28.834256 9269 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260924 04:59:28.834635 9269 webserver.cc:460] Webserver started at http://127.0.98.2:34853/ using document root <none> and password file <none>
I20260924 04:59:28.834848 9269 fs_manager.cc:359] Metadata directory not provided
I20260924 04:59:28.834908 9269 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:59:28.836166 9269 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 04:59:28.836671 9287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:28.836826 9269 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:28.836896 9269 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/ts-1/wal
uuid: "330ef213c92640339e66f9d2dce79b24"
format_stamp: "Formatted at 2026-09-24 04:59:28 on dist-test-slave-70st"
server_key: "b3bd291ae8079c218e04324335f0980a"
server_key_iv: "20a05c14c9c2c33b218c99e730c044cf"
server_key_version: "encryptionkey@0"
I20260924 04:59:28.837210 9269 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:59:28.858891 9269 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:59:28.861871 9269 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:59:28.862017 9269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:59:28.862254 9269 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:59:28.862632 9269 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:59:28.862674 9269 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:28.862703 9269 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:59:28.862726 9269 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:28.869660 9269 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36057
I20260924 04:59:28.869705 9400 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36057 every 8 connection(s)
I20260924 04:59:28.870118 9269 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:59:28.873858 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 9269
I20260924 04:59:28.874166 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:39663
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:37517
--webserver_port=42627
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 04:59:28 dist-test-slave-70st krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:59:28 dist-test-slave-70st krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 04:59:28.877874 9271 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:59:28.870804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47317 (local address 127.0.98.62:37517)
0924 04:59:28.870845 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:28.870847 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:28.871092 (+ 245us) server_negotiation.cc:407] Connection header received
0924 04:59:28.871634 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:28.871635 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:28.871652 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:28.871729 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:28.872589 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:28.873353 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:28.874588 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:28.874791 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:28.874847 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:28.877818 (+ 2971us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:47317: BlockingRecv error: recv got EOF from 127.0.98.2:47317 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 04:59:28.877934 9403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:28.870829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.2:47317)
0924 04:59:28.870982 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:28.871016 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:28.871527 (+ 511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:28.871783 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:28.871786 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:28.872315 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:28.872502 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:28.872507 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:28.873532 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:28.873533 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:28.874508 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:28.874511 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:28.874856 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:28.874857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:28.874981 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:28.875476 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:28.875484 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:28.877743 (+ 2259us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":87,"threads_started":1}
W20260924 04:59:28.878175 9401 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37517 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:37517: Client connection negotiation failed: client connection to 127.0.98.62:37517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 24 04:59:28 dist-test-slave-70st krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:59:28 dist-test-slave-70st krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:59:28.882613 9403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:28.878485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.2:33547)
0924 04:59:28.878525 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:28.878534 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:28.878565 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:28.878683 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:28.878683 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:28.878806 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:28.878846 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:28.878848 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:28.879646 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:28.879646 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:28.880187 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:28.880189 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:28.880600 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:28.880601 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:28.880663 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:28.881050 (+ 387us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:28.881056 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:28.882553 (+ 1497us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":27}
W20260924 04:59:28.882691 9271 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:59:28.878545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33547 (local address 127.0.98.62:37517)
0924 04:59:28.878598 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:28.878600 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:28.878606 (+ 6us) server_negotiation.cc:407] Connection header received
0924 04:59:28.878627 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:28.878628 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:28.878636 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:28.878679 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:28.878907 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:28.879589 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:28.880256 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:28.880542 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:28.880602 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:28.882635 (+ 2033us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:33547: BlockingRecv error: recv got EOF from 127.0.98.2:33547 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 24 04:59:28 dist-test-slave-70st krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:59:28 dist-test-slave-70st krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:59:28.886601 9403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:28.883038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.2:58827)
0924 04:59:28.883076 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:28.883084 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:28.883114 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:28.883219 (+ 105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:28.883219 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:28.883333 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:28.883358 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:28.883359 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:28.883911 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:28.883912 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:28.884385 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:28.884387 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:28.884674 (+ 287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:28.884675 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:28.884742 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:28.885081 (+ 339us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:28.885085 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:28.886542 (+ 1457us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":24}
W20260924 04:59:28.886662 9271 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:59:28.883106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58827 (local address 127.0.98.62:37517)
0924 04:59:28.883141 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:28.883143 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:28.883150 (+ 7us) server_negotiation.cc:407] Connection header received
0924 04:59:28.883167 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:28.883168 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:28.883176 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:28.883217 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:28.883414 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:28.883851 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:28.884447 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:28.884623 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:28.884672 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:28.886610 (+ 1938us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:58827: BlockingRecv error: recv got EOF from 127.0.98.2:58827 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260924 04:59:28.886938 9401 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 04:59:28.955036 9404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:59:28.955225 9404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:59:28.955245 9404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:59:28.956331 9404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:59:28.956369 9404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:59:28.956393 9404 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:59:28.957731 9404 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:39663
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=42627
--tserver_master_addrs=127.0.98.62:37517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9404
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 04:59:28.957902 9404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:59:28.958062 9404 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:59:28.962641 9409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:59:28.962631 9412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:59:28.962616 9410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:59:28.962702 9404 server_base.cc:1034] running on GCE node
I20260924 04:59:28.962980 9404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:59:28.963189 9404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:59:28.964325 9404 hybrid_clock.cc:648] HybridClock initialized: now 1790225968964312 us; error 30 us; skew 500 ppm
Sep 24 04:59:28 dist-test-slave-70st krb5kdc[8536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225968, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:59:28.966610 9404 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260924 04:59:28.967022 9404 webserver.cc:460] Webserver started at http://127.0.98.3:42627/ using document root <none> and password file <none>
I20260924 04:59:28.967223 9404 fs_manager.cc:359] Metadata directory not provided
I20260924 04:59:28.967283 9404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:59:28.968552 9404 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 04:59:28.969072 9419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:28.969235 9404 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:28.969306 9404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/ts-2/wal
uuid: "e4b5732ab3c6431bb161ee9a4aff6bda"
format_stamp: "Formatted at 2026-09-24 04:59:28 on dist-test-slave-70st"
server_key: "10abcecf4c39c297f1f816e44042e237"
server_key_iv: "8bc7aabf53250f14d5e8698a7c0586b7"
server_key_version: "encryptionkey@0"
I20260924 04:59:28.969668 9404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:59:28.985977 9404 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:59:28.987843 9404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:59:28.987974 9404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:59:28.988193 9404 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:59:28.988482 9404 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:59:28.988526 9404 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:28.988554 9404 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:59:28.988570 9404 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:28.993844 9404 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39663
I20260924 04:59:28.993893 9532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39663 every 8 connection(s)
I20260924 04:59:28.994195 9404 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790225929848825-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 04:59:28 dist-test-slave-70st krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:59:28.999962 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 9404
Sep 24 04:59:28 dist-test-slave-70st krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 04:59:29.000849 9271 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:59:28.994830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47053 (local address 127.0.98.62:37517)
0924 04:59:28.994874 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:28.994876 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:28.995083 (+ 207us) server_negotiation.cc:407] Connection header received
0924 04:59:28.995527 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:28.995528 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:28.995538 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:28.995588 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:28.996446 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:28.996917 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:28.997898 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:28.998085 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:28.998136 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:29.000794 (+ 2658us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:47053: BlockingRecv error: recv got EOF from 127.0.98.3:47053 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 04:59:29.000986 9535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:28.994825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.3:47053)
0924 04:59:28.994972 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:28.995006 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:28.995454 (+ 448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:28.995641 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:28.995644 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:28.996178 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:28.996371 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:28.996375 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:28.996981 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:28.996981 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:28.997810 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:28.997813 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:28.998143 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:28.998144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:28.998235 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:28.998661 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:28.998668 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:29.000719 (+ 2051us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":75,"threads_started":1}
W20260924 04:59:29.001186 9533 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37517 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:37517: Client connection negotiation failed: client connection to 127.0.98.62:37517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 24 04:59:29 dist-test-slave-70st krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:59:29 dist-test-slave-70st krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:59:29.005754 9535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:29.001387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.3:46977)
0924 04:59:29.001442 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:29.001452 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:29.001484 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:29.001582 (+ 98us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:29.001582 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:29.001713 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:29.001759 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:29.001761 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:29.002316 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:29.002316 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:29.003076 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:29.003078 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:29.003447 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:29.003448 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:29.003507 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:29.003967 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:29.003972 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:29.005690 (+ 1718us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":20}
W20260924 04:59:29.005827 9271 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:59:29.001452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46977 (local address 127.0.98.62:37517)
0924 04:59:29.001487 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:29.001489 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:29.001515 (+ 26us) server_negotiation.cc:407] Connection header received
0924 04:59:29.001536 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:29.001536 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:29.001543 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:29.001577 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:29.001819 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:29.002254 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:29.003143 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:29.003392 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:29.003443 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:29.005776 (+ 2333us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:46977: BlockingRecv error: recv got EOF from 127.0.98.3:46977 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 04:59:29.007767 9272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:29.000449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47518 (local address 127.0.98.62:37517)
0924 04:59:29.000528 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:29.000530 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:29.000538 (+ 8us) server_negotiation.cc:407] Connection header received
0924 04:59:29.000592 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:29.000592 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:29.000603 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:29.000659 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:29.001382 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:29.002050 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:29.003071 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:29.003360 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:29.003423 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:29.004686 (+ 1263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:29.004691 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:29.004691 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:59:29.004703 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:29.006479 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:29.006892 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:29.006895 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:29.006896 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:29.006984 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:29.007148 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:29.007149 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:29.007149 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:29.007366 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:29.007499 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:29.007596 (+ 97us) server_negotiation.cc:299] Negotiation successful
0924 04:59:29.007697 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
Sep 24 04:59:29 dist-test-slave-70st krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:59:29 dist-test-slave-70st krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:59:29.010529 9535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:29.006699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.3:54871)
0924 04:59:29.006752 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:29.006763 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:29.006805 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:29.006913 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:29.006913 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:29.007024 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:29.007049 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:29.007050 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:29.007642 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:29.007642 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:29.008303 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:29.008306 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:29.008696 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:29.008697 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:29.008733 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:29.009066 (+ 333us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:29.009070 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:29.010487 (+ 1417us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":36}
W20260924 04:59:29.010646 9533 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 04:59:29.010600 9271 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:59:29.006728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54871 (local address 127.0.98.62:37517)
0924 04:59:29.006762 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:29.006764 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:29.006793 (+ 29us) server_negotiation.cc:407] Connection header received
0924 04:59:29.006861 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:29.006862 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:29.006871 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:29.006911 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:29.007136 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:29.007582 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:29.008399 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:29.008639 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:29.008689 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:29.010552 (+ 1863us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:54871: BlockingRecv error: recv got EOF from 127.0.98.3:54871 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
Sep 24 04:59:29 dist-test-slave-70st krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:59:29 dist-test-slave-70st krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:59:29.765444 9537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:29.761101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.1:58335)
0924 04:59:29.761270 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:29.761281 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:29.761342 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:29.761491 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:29.761492 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:29.761642 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:29.761692 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:29.761694 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:29.762323 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:29.762324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:29.762887 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:29.762889 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:29.763183 (+ 294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:29.763183 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:29.763220 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:29.763647 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:29.763651 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:29.765336 (+ 1685us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":71,"threads_started":1}
W20260924 04:59:29.765551 9538 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:59:29.761163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58335 (local address 127.0.98.62:37517)
0924 04:59:29.761297 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:29.761298 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:29.761326 (+ 28us) server_negotiation.cc:407] Connection header received
0924 04:59:29.761400 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:29.761401 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:29.761410 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:29.761486 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:29.761749 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:29.762229 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:29.762954 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:29.763132 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:29.763176 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:29.765482 (+ 2306us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:58335: BlockingRecv error: recv got EOF from 127.0.98.1:58335 (error 108)
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":60,"threads_started":1}
Sep 24 04:59:29 dist-test-slave-70st krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:59:29 dist-test-slave-70st krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:59:29.891997 9539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:29.887352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.2:40007)
0924 04:59:29.887493 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:29.887506 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:29.887586 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:29.887713 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:29.887714 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:29.887887 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:29.887933 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:29.887935 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:29.888579 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:29.888580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:29.889127 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:29.889128 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:29.889483 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:29.889485 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:29.889539 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:29.889977 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:29.889983 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:29.891906 (+ 1923us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":67,"threads_started":1}
W20260924 04:59:29.892252 9538 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:59:29.887440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40007 (local address 127.0.98.62:37517)
0924 04:59:29.887500 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:29.887502 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:29.887568 (+ 66us) server_negotiation.cc:407] Connection header received
0924 04:59:29.887647 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:29.887648 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:29.887658 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:29.887700 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:29.888005 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:29.888498 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:29.889213 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:29.889398 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:29.889485 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:29.892197 (+ 2712us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:40007: BlockingRecv error: recv got EOF from 127.0.98.2:40007 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 24 04:59:30 dist-test-slave-70st krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:59:30 dist-test-slave-70st krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:59:30.015401 9540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:30.011026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.3:37025)
0924 04:59:30.011171 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:30.011182 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:30.011249 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:30.011369 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:30.011370 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:30.011526 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:30.011576 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:30.011577 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:30.012292 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:30.012293 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:30.012830 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:30.012831 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:30.013165 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:30.013165 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:30.013199 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:30.013680 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:30.013684 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:30.015330 (+ 1646us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":58,"threads_started":1}
W20260924 04:59:30.015491 9538 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:59:30.011122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37025 (local address 127.0.98.62:37517)
0924 04:59:30.011186 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:30.011188 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:30.011242 (+ 54us) server_negotiation.cc:407] Connection header received
0924 04:59:30.011296 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:30.011297 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:30.011308 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:30.011359 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:30.011668 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:30.012226 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:30.012919 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:30.013107 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:30.013164 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:30.015425 (+ 2261us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:37025: BlockingRecv error: recv got EOF from 127.0.98.3:37025 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 24 04:59:30 dist-test-slave-70st krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:59:30 dist-test-slave-70st krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:59:30.777029 9541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:30.766060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.1:58491)
0924 04:59:30.766171 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:30.766181 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:30.766230 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:30.766367 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:30.766368 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:30.766522 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:30.766569 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:30.766570 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:30.767168 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:30.767169 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:30.767701 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:30.767702 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:30.768031 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:30.768031 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:30.768069 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:30.775121 (+ 7052us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:30.775129 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:30.776968 (+ 1839us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":61,"threads_started":1}
W20260924 04:59:30.777194 9542 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:59:30.766147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58491 (local address 127.0.98.62:37517)
0924 04:59:30.766251 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:30.766253 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:30.766262 (+ 9us) server_negotiation.cc:407] Connection header received
0924 04:59:30.766291 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:30.766292 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:30.766302 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:30.766355 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:30.766626 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:30.767113 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:30.767776 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:30.767972 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:30.768028 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:30.777111 (+ 9083us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:58491: BlockingRecv error: recv got EOF from 127.0.98.1:58491 (error 108)
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":50,"threads_started":1}
Sep 24 04:59:30 dist-test-slave-70st krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:59:30 dist-test-slave-70st krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:59:30.897536 9543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:30.892486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.2:41641)
0924 04:59:30.892610 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:30.892620 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:30.892685 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:30.892816 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:30.892817 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:30.892977 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:30.893024 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:30.893026 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:30.893710 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:30.893711 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:30.894452 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:30.894455 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:30.894896 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:30.894897 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:30.894965 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:30.895454 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:30.895460 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:30.897469 (+ 2009us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":64,"threads_started":1}
W20260924 04:59:30.897598 9542 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:59:30.892603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41641 (local address 127.0.98.62:37517)
0924 04:59:30.892664 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:30.892666 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:30.892675 (+ 9us) server_negotiation.cc:407] Connection header received
0924 04:59:30.892735 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:30.892735 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:30.892746 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:30.892801 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:30.893076 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:30.893642 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:30.894560 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:30.894823 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:30.894883 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:30.897545 (+ 2662us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:41641: BlockingRecv error: recv got EOF from 127.0.98.2:41641 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 24 04:59:31 dist-test-slave-70st krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:59:31 dist-test-slave-70st krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:59:31.021543 9544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:31.015869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.3:54691)
0924 04:59:31.016012 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:31.016025 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:31.016097 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:31.016271 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:31.016272 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:31.016454 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:31.016504 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:31.016505 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:31.017177 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:31.017178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:31.017930 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:31.017933 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:31.018427 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:31.018428 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:31.018487 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:31.019155 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:31.019161 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:31.021462 (+ 2301us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":63,"threads_started":1}
W20260924 04:59:31.021613 9542 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:59:31.015989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54691 (local address 127.0.98.62:37517)
0924 04:59:31.016040 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:31.016042 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:31.016077 (+ 35us) server_negotiation.cc:407] Connection header received
0924 04:59:31.016173 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:31.016174 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:31.016186 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:31.016261 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:31.016605 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:31.017097 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:31.018038 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:31.018350 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:31.018414 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:31.021551 (+ 3137us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:54691: BlockingRecv error: recv got EOF from 127.0.98.3:54691 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Sep 24 04:59:31 dist-test-slave-70st krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:59:31 dist-test-slave-70st krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:59:31.782270 9545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:31.777607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.1:50185)
0924 04:59:31.777767 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:31.777782 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:31.777863 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:31.778013 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:31.778015 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:31.778232 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:31.778298 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:31.778302 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:31.779005 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:31.779006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:31.779563 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:31.779565 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:31.779924 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:31.779925 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:31.779979 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:31.780405 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:31.780409 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:31.782197 (+ 1788us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":61,"threads_started":1}
W20260924 04:59:31.782379 9546 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:59:31.777747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50185 (local address 127.0.98.62:37517)
0924 04:59:31.777854 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:31.777856 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:31.777866 (+ 10us) server_negotiation.cc:407] Connection header received
0924 04:59:31.777924 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:31.777925 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:31.777938 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:31.777996 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:31.778405 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:31.778932 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:31.779649 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:31.779840 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:31.779892 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:31.782308 (+ 2416us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:50185: BlockingRecv error: recv got EOF from 127.0.98.1:50185 (error 108)
Metrics: {"server-negotiator.queue_time_us":83,"thread_start_us":40,"threads_started":1}
Sep 24 04:59:31 dist-test-slave-70st krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:59:31 dist-test-slave-70st krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:59:31.903220 9547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:31.898077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.2:46915)
0924 04:59:31.898296 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:31.898309 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:31.898390 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:31.898585 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:31.898586 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:31.898769 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:31.898825 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:31.898827 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:31.899771 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:31.899772 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:31.900427 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:31.900429 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:31.900807 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:31.900808 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:31.900860 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:31.901337 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:31.901342 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:31.903151 (+ 1809us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":70,"threads_started":1}
W20260924 04:59:31.903334 9546 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:59:31.898209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46915 (local address 127.0.98.62:37517)
0924 04:59:31.898306 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:31.898308 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:31.898370 (+ 62us) server_negotiation.cc:407] Connection header received
0924 04:59:31.898451 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:31.898452 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:31.898465 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:31.898529 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:31.898914 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:31.899680 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:31.900516 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:31.900738 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:31.900803 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:31.903262 (+ 2459us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:46915: BlockingRecv error: recv got EOF from 127.0.98.2:46915 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 24 04:59:32 dist-test-slave-70st krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:59:32 dist-test-slave-70st krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:59:32.026531 9548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:32.022105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.3:59169)
0924 04:59:32.022240 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:32.022250 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:32.022317 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:32.022441 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:32.022441 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:32.022630 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:32.022676 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:32.022678 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:32.023285 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:32.023285 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:32.023805 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:32.023806 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:32.024141 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:32.024142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:32.024183 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:32.024662 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:32.024667 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:32.026446 (+ 1779us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":62,"threads_started":1}
W20260924 04:59:32.026585 9546 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:59:32.022197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59169 (local address 127.0.98.62:37517)
0924 04:59:32.022245 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:32.022248 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:32.022326 (+ 78us) server_negotiation.cc:407] Connection header received
0924 04:59:32.022369 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:32.022369 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:32.022380 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:32.022427 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:32.022747 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:32.023228 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:32.023892 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:32.024079 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:32.024133 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:32.026531 (+ 2398us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:59169: BlockingRecv error: recv got EOF from 127.0.98.3:59169 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Sep 24 04:59:32 dist-test-slave-70st krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:59:32 dist-test-slave-70st krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:59:32.787214 9549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:32.782787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.1:58337)
0924 04:59:32.782937 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:32.782948 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:32.783010 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:32.783144 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:32.783144 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:32.783296 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:32.783344 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:32.783346 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:32.783972 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:32.783973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:32.784495 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:32.784497 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:32.784831 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:32.784832 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:32.784882 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:32.785342 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:32.785347 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:32.787156 (+ 1809us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":58,"threads_started":1}
W20260924 04:59:32.787349 9550 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:59:32.782908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58337 (local address 127.0.98.62:37517)
0924 04:59:32.783031 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:32.783033 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:32.783040 (+ 7us) server_negotiation.cc:407] Connection header received
0924 04:59:32.783064 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:32.783065 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:32.783074 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:32.783131 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:32.783402 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:32.783908 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:32.784560 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:32.784744 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:32.784790 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:32.787243 (+ 2453us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:58337: BlockingRecv error: recv got EOF from 127.0.98.1:58337 (error 108)
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":62,"threads_started":1}
Sep 24 04:59:32 dist-test-slave-70st krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:59:32 dist-test-slave-70st krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:59:32.908123 9551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:32.903658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.2:60243)
0924 04:59:32.903814 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:32.903827 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:32.903919 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:32.904032 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:32.904032 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:32.904178 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:32.904224 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:32.904225 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:32.904883 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:32.904884 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:32.905486 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:32.905489 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:32.905922 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:32.905923 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:32.905982 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:32.906413 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:32.906418 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:32.908062 (+ 1644us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":63,"threads_started":1}
W20260924 04:59:32.908206 9550 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:59:32.903786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60243 (local address 127.0.98.62:37517)
0924 04:59:32.903827 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:32.903830 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:32.903898 (+ 68us) server_negotiation.cc:407] Connection header received
0924 04:59:32.903969 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:32.903970 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:32.903980 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:32.904027 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:32.904273 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:32.904820 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:32.905570 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:32.905844 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:32.905902 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:32.908149 (+ 2247us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:60243: BlockingRecv error: recv got EOF from 127.0.98.2:60243 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
Sep 24 04:59:33 dist-test-slave-70st krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:59:33 dist-test-slave-70st krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:59:33.031359 9552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:33.027002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.3:48965)
0924 04:59:33.027142 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:33.027152 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:33.027207 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:33.027318 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:33.027319 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:33.027468 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:33.027516 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:33.027517 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:33.028191 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:33.028192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:33.028725 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:33.028727 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:33.029086 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:33.029087 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:33.029130 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:33.029592 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:33.029596 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:33.031298 (+ 1702us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":73,"threads_started":1}
W20260924 04:59:33.031493 9550 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:59:33.027062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48965 (local address 127.0.98.62:37517)
0924 04:59:33.027104 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:33.027106 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:33.027192 (+ 86us) server_negotiation.cc:407] Connection header received
0924 04:59:33.027250 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:33.027250 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:33.027260 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:33.027313 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:33.027575 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:33.028103 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:33.028818 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:33.029014 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:33.029087 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:33.031423 (+ 2336us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:48965: BlockingRecv error: recv got EOF from 127.0.98.3:48965 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
Sep 24 04:59:33 dist-test-slave-70st krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:59:33 dist-test-slave-70st krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:59:33.792004 9553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:33.787704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.1:40279)
0924 04:59:33.787870 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:33.787879 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:33.787932 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:33.788069 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:33.788069 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:33.788223 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:33.788268 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:33.788269 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:33.788890 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:33.788890 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:33.789491 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:33.789493 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:33.789826 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:33.789827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:33.789882 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:33.790335 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:33.790340 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:33.791946 (+ 1606us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":106,"threads_started":1}
W20260924 04:59:33.792096 9554 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:59:33.787785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40279 (local address 127.0.98.62:37517)
0924 04:59:33.787915 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:33.787917 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:33.787927 (+ 10us) server_negotiation.cc:407] Connection header received
0924 04:59:33.787987 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:33.787987 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:33.787997 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:33.788058 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:33.788328 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:33.788823 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:33.789552 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:33.789771 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:33.789828 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:33.792030 (+ 2202us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:40279: BlockingRecv error: recv got EOF from 127.0.98.1:40279 (error 108)
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":52,"threads_started":1}
Sep 24 04:59:33 dist-test-slave-70st krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:59:33 dist-test-slave-70st krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:59:33.912941 9555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:33.908573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.2:60781)
0924 04:59:33.908699 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:33.908710 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:33.908767 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:33.908888 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:33.908888 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:33.909060 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:33.909103 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:33.909105 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:33.909776 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:33.909778 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:33.910353 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:33.910356 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:33.910678 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:33.910679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:33.910731 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:33.911157 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:33.911162 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:33.912851 (+ 1689us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":65,"threads_started":1}
W20260924 04:59:33.913043 9554 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:59:33.908697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60781 (local address 127.0.98.62:37517)
0924 04:59:33.908777 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:33.908778 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:33.908788 (+ 10us) server_negotiation.cc:407] Connection header received
0924 04:59:33.908815 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:33.908816 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:33.908827 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:33.908881 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:33.909173 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:33.909706 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:33.910435 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:33.910618 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:33.910672 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:33.912973 (+ 2301us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:60781: BlockingRecv error: recv got EOF from 127.0.98.2:60781 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 24 04:59:34 dist-test-slave-70st krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:59:34 dist-test-slave-70st krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:59:34.036159 9556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:34.031829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.3:42415)
0924 04:59:34.031946 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:34.031957 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:34.032013 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:34.032146 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:34.032147 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:34.032315 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:34.032365 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:34.032367 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:34.032998 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:34.032998 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:34.033621 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:34.033623 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:34.033944 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:34.033945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:34.033980 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:34.034369 (+ 389us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:34.034373 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:34.036077 (+ 1704us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":73,"threads_started":1}
W20260924 04:59:34.036211 9554 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:59:34.031902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42415 (local address 127.0.98.62:37517)
0924 04:59:34.031953 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:34.031955 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:34.031998 (+ 43us) server_negotiation.cc:407] Connection header received
0924 04:59:34.032054 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:34.032054 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:34.032065 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:34.032121 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:34.032422 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:34.032929 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:34.033688 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:34.033883 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:34.033943 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:34.036161 (+ 2218us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:42415: BlockingRecv error: recv got EOF from 127.0.98.3:42415 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Sep 24 04:59:34 dist-test-slave-70st krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:59:34 dist-test-slave-70st krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:59:34.797134 9557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:34.792469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.1:57239)
0924 04:59:34.792627 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:34.792639 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:34.792697 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:34.792845 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:34.792846 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:34.793003 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:34.793049 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:34.793050 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:34.793711 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:34.793713 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:34.794250 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:34.794252 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:34.794687 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:34.794688 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:34.794747 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:34.795242 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:34.795247 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:34.797076 (+ 1829us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":74,"threads_started":1}
W20260924 04:59:34.797302 9558 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:59:34.792539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57239 (local address 127.0.98.62:37517)
0924 04:59:34.792651 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:34.792652 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:34.792684 (+ 32us) server_negotiation.cc:407] Connection header received
0924 04:59:34.792760 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:34.792760 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:34.792770 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:34.792830 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:34.793102 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:34.793621 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:34.794395 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:34.794595 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:34.794659 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:34.797182 (+ 2523us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:57239: BlockingRecv error: recv got EOF from 127.0.98.1:57239 (error 108)
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":59,"threads_started":1}
Sep 24 04:59:34 dist-test-slave-70st krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:59:34 dist-test-slave-70st krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:59:34.917747 9559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:34.913493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.2:56833)
0924 04:59:34.913609 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:34.913621 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:34.913707 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:34.913854 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:34.913855 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:34.914016 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:34.914061 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:34.914063 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:34.914724 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:34.914725 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:34.915257 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:34.915259 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:34.915589 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:34.915589 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:34.915643 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:34.916055 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:34.916059 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:34.917689 (+ 1630us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":61,"threads_started":1}
W20260924 04:59:34.917846 9558 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:59:34.913556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56833 (local address 127.0.98.62:37517)
0924 04:59:34.913620 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:34.913623 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:34.913685 (+ 62us) server_negotiation.cc:407] Connection header received
0924 04:59:34.913764 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:34.913765 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:34.913776 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:34.913820 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:34.914133 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:34.914647 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:34.915335 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:34.915531 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:34.915586 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:34.917780 (+ 2194us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:56833: BlockingRecv error: recv got EOF from 127.0.98.2:56833 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 24 04:59:35 dist-test-slave-70st krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:59:35 dist-test-slave-70st krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:59:35.041231 9560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:35.036750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.3:36529)
0924 04:59:35.036884 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:35.036895 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:35.036967 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:35.037099 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:35.037099 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:35.037294 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:35.037341 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:35.037343 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:35.038053 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:35.038055 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:35.038591 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:35.038593 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:35.038901 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:35.038902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:35.038953 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:35.039410 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:35.039416 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:35.041150 (+ 1734us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":60,"threads_started":1}
W20260924 04:59:35.041363 9558 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:59:35.036843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36529 (local address 127.0.98.62:37517)
0924 04:59:35.036903 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:35.036905 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:35.036962 (+ 57us) server_negotiation.cc:407] Connection header received
0924 04:59:35.037030 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:35.037031 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:35.037045 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:35.037086 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:35.037408 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:35.037977 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:35.038662 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:35.038844 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:35.038906 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:35.041264 (+ 2358us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:36529: BlockingRecv error: recv got EOF from 127.0.98.3:36529 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Sep 24 04:59:35 dist-test-slave-70st krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:59:35 dist-test-slave-70st krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:59:35.810555 9561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:35.797678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.1:37117)
0924 04:59:35.797839 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:35.797850 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:35.797912 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:35.798075 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:35.798077 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:35.798235 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:35.798284 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:35.798286 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:35.799059 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:35.799060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:35.799770 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:35.799772 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:35.800107 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:35.800108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:35.800158 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:35.808589 (+ 8431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:35.808594 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:35.810477 (+ 1883us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":62,"threads_started":1}
W20260924 04:59:35.810647 9562 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:59:35.797779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37117 (local address 127.0.98.62:37517)
0924 04:59:35.797891 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:35.797893 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:35.797904 (+ 11us) server_negotiation.cc:407] Connection header received
0924 04:59:35.797960 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:35.797960 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:35.797970 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:35.798036 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:35.798393 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:35.798980 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:35.799843 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:35.800041 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:35.800096 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:35.810585 (+ 10489us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:37117: BlockingRecv error: recv got EOF from 127.0.98.1:37117 (error 108)
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":54,"threads_started":1}
Sep 24 04:59:35 dist-test-slave-70st krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:59:35 dist-test-slave-70st krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:59:35.922547 9563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:35.918154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.2:54541)
0924 04:59:35.918293 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:35.918304 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:35.918369 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:35.918523 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:35.918524 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:35.918687 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:35.918733 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:35.918734 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:35.919392 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:35.919393 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:35.919922 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:35.919923 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:35.920239 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:35.920240 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:35.920290 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:35.920713 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:35.920718 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:35.922467 (+ 1749us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":69,"threads_started":1}
W20260924 04:59:35.922621 9562 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:59:35.918190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54541 (local address 127.0.98.62:37517)
0924 04:59:35.918249 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:35.918251 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:35.918358 (+ 107us) server_negotiation.cc:407] Connection header received
0924 04:59:35.918450 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:35.918450 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:35.918461 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:35.918510 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:35.918799 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:35.919310 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:35.919994 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:35.920182 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:35.920236 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:35.922568 (+ 2332us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:54541: BlockingRecv error: recv got EOF from 127.0.98.2:54541 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 24 04:59:36 dist-test-slave-70st krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:59:36 dist-test-slave-70st krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:59:36.046025 9564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:36.041772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.3:34791)
0924 04:59:36.041914 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:36.041925 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:36.041991 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:36.042107 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:36.042108 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:36.042261 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:36.042320 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:36.042322 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:36.042962 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:36.042963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:36.043520 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:36.043522 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:36.043870 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:36.043871 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:36.043916 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:36.044328 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:36.044332 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:36.045965 (+ 1633us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":70,"threads_started":1}
W20260924 04:59:36.046113 9562 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:59:36.041853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34791 (local address 127.0.98.62:37517)
0924 04:59:36.041914 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:36.041916 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:36.041982 (+ 66us) server_negotiation.cc:407] Connection header received
0924 04:59:36.042034 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:36.042035 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:36.042047 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:36.042094 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:36.042389 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:36.042892 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:36.043603 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:36.043792 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:36.043854 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:36.046048 (+ 2194us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:34791: BlockingRecv error: recv got EOF from 127.0.98.3:34791 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 24 04:59:36 dist-test-slave-70st krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:59:36 dist-test-slave-70st krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:59:36.815721 9565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:36.811063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.1:46123)
0924 04:59:36.811210 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:36.811221 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:36.811276 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:36.811531 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:36.811532 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:36.811706 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:36.811760 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:36.811762 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:36.812426 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:36.812427 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:36.812964 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:36.812966 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:36.813339 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:36.813340 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:36.813391 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:36.813919 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:36.813923 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:36.815661 (+ 1738us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":59,"threads_started":1}
W20260924 04:59:36.815876 9566 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:59:36.811216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46123 (local address 127.0.98.62:37517)
0924 04:59:36.811371 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:36.811373 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:36.811382 (+ 9us) server_negotiation.cc:407] Connection header received
0924 04:59:36.811408 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:36.811409 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:36.811419 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:36.811477 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:36.811830 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:36.812335 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:36.813061 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:36.813261 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:36.813323 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:36.815800 (+ 2477us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:46123: BlockingRecv error: recv got EOF from 127.0.98.1:46123 (error 108)
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":87,"threads_started":1}
Sep 24 04:59:36 dist-test-slave-70st krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:59:36 dist-test-slave-70st krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:59:36.929311 9567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:36.922994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.2:37015)
0924 04:59:36.923127 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:36.923137 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:36.923197 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:36.924403 (+ 1206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:36.924405 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:36.924612 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:36.924673 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:36.924675 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:36.925822 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:36.925823 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:36.926557 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:36.926560 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:36.926922 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:36.926922 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:36.926974 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:36.927437 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:36.927442 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:36.929246 (+ 1804us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":66,"threads_started":1}
W20260924 04:59:36.929399 9566 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:59:36.923036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37015 (local address 127.0.98.62:37517)
0924 04:59:36.923082 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:36.923084 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:36.924256 (+ 1172us) server_negotiation.cc:407] Connection header received
0924 04:59:36.924291 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:36.924292 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:36.924306 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:36.924371 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:36.924762 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:36.925738 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:36.926645 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:36.926856 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:36.926917 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:36.929340 (+ 2423us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:37015: BlockingRecv error: recv got EOF from 127.0.98.2:37015 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Sep 24 04:59:37 dist-test-slave-70st krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:59:37 dist-test-slave-70st krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:59:37.050913 9568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:37.046497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.3:35835)
0924 04:59:37.046625 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:37.046634 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:37.046682 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:37.046813 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:37.046814 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:37.046982 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:37.047027 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:37.047029 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:37.047671 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:37.047672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:37.048183 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:37.048184 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:37.048557 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:37.048558 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:37.048600 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:37.049021 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:37.049026 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:37.050831 (+ 1805us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":68,"threads_started":1}
W20260924 04:59:37.050992 9566 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:59:37.046589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35835 (local address 127.0.98.62:37517)
0924 04:59:37.046656 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:37.046658 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:37.046671 (+ 13us) server_negotiation.cc:407] Connection header received
0924 04:59:37.046733 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:37.046733 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:37.046744 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:37.046803 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:37.047097 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:37.047602 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:37.048266 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:37.048488 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:37.048546 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:37.050938 (+ 2392us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:35835: BlockingRecv error: recv got EOF from 127.0.98.3:35835 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 24 04:59:37 dist-test-slave-70st krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:59:37 dist-test-slave-70st krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:59:37.820452 9569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:37.816219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.1:35737)
0924 04:59:37.816342 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:37.816353 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:37.816400 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:37.816610 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:37.816611 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:37.816763 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:37.816810 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:37.816812 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:37.817485 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:37.817485 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:37.818010 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:37.818012 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:37.818321 (+ 309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:37.818322 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:37.818355 (+ 33us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:37.818748 (+ 393us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:37.818752 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:37.820393 (+ 1641us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":67,"threads_started":1}
W20260924 04:59:37.820595 9570 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:59:37.816327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35737 (local address 127.0.98.62:37517)
0924 04:59:37.816494 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:37.816496 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:37.816504 (+ 8us) server_negotiation.cc:407] Connection header received
0924 04:59:37.816527 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:37.816527 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:37.816537 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:37.816602 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:37.816865 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:37.817380 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:37.818075 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:37.818268 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:37.818322 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:37.820513 (+ 2191us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:35737: BlockingRecv error: recv got EOF from 127.0.98.1:35737 (error 108)
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":84,"threads_started":1}
Sep 24 04:59:37 dist-test-slave-70st krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:59:37 dist-test-slave-70st krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:59:37.934810 9571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:37.929776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.2:34511)
0924 04:59:37.929906 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:37.929916 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:37.929983 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:37.930103 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:37.930104 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:37.930266 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:37.930314 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:37.930315 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:37.930956 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:37.930956 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:37.931498 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:37.931499 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:37.932008 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:37.932009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:37.932070 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:37.932548 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:37.932553 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:37.934746 (+ 2193us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":66,"threads_started":1}
W20260924 04:59:37.934929 9570 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:59:37.929844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34511 (local address 127.0.98.62:37517)
0924 04:59:37.929904 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:37.929907 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:37.929963 (+ 56us) server_negotiation.cc:407] Connection header received
0924 04:59:37.930035 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:37.930036 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:37.930047 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:37.930089 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:37.930380 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:37.930882 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:37.931589 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:37.931934 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:37.931995 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:37.934862 (+ 2867us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:34511: BlockingRecv error: recv got EOF from 127.0.98.2:34511 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 24 04:59:38 dist-test-slave-70st krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:59:38 dist-test-slave-70st krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:59:38.056023 9572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:38.051441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.3:33913)
0924 04:59:38.051602 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:38.051611 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:38.051668 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:38.051793 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:38.051809 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:38.051972 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:38.052016 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:38.052017 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:38.052688 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:38.052689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:38.053258 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:38.053260 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:38.053623 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:38.053624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:38.053677 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:38.054174 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:38.054179 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:38.055945 (+ 1766us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":82,"threads_started":1}
W20260924 04:59:38.056102 9570 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:59:38.051506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33913 (local address 127.0.98.62:37517)
0924 04:59:38.051555 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:38.051557 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:38.051663 (+ 106us) server_negotiation.cc:407] Connection header received
0924 04:59:38.051716 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:38.051717 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:38.051730 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:38.051774 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:38.052084 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:38.052605 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:38.053346 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:38.053563 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:38.053619 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:38.056045 (+ 2426us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:33913: BlockingRecv error: recv got EOF from 127.0.98.3:33913 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Sep 24 04:59:38 dist-test-slave-70st krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:59:38 dist-test-slave-70st krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:59:38.825668 9573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:38.820966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.1:58813)
0924 04:59:38.821131 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:38.821144 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:38.821219 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:38.821385 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:38.821386 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:38.821575 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:38.821642 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:38.821644 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:38.822519 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:38.822520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:38.823058 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:38.823059 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:38.823369 (+ 310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:38.823370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:38.823405 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:38.823813 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:38.823817 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:38.825592 (+ 1775us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":70,"threads_started":1}
W20260924 04:59:38.825744 9574 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:59:38.821083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58813 (local address 127.0.98.62:37517)
0924 04:59:38.821209 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:38.821210 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:38.821220 (+ 10us) server_negotiation.cc:407] Connection header received
0924 04:59:38.821269 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:38.821269 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:38.821279 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:38.821339 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:38.821742 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:38.822438 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:38.823122 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:38.823315 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:38.823367 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:38.825683 (+ 2316us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:58813: BlockingRecv error: recv got EOF from 127.0.98.1:58813 (error 108)
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":67,"threads_started":1}
Sep 24 04:59:38 dist-test-slave-70st krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:59:38 dist-test-slave-70st krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:59:38.939572 9575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:38.935243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.2:34867)
0924 04:59:38.935361 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:38.935370 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:38.935433 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:38.935558 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:38.935559 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:38.935707 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:38.935754 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:38.935755 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:38.936371 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:38.936372 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:38.936900 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:38.936902 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:38.937185 (+ 283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:38.937186 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:38.937224 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:38.937652 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:38.937656 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:38.939494 (+ 1838us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":57,"threads_started":1}
W20260924 04:59:38.939651 9574 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:59:38.935308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34867 (local address 127.0.98.62:37517)
0924 04:59:38.935362 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:38.935364 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:38.935425 (+ 61us) server_negotiation.cc:407] Connection header received
0924 04:59:38.935486 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:38.935487 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:38.935496 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:38.935549 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:38.935802 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:38.936282 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:38.936958 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:38.937137 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:38.937180 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:38.939599 (+ 2419us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:34867: BlockingRecv error: recv got EOF from 127.0.98.2:34867 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Sep 24 04:59:39 dist-test-slave-70st krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:59:39 dist-test-slave-70st krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:59:39.061151 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:39.056557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.3:55677)
0924 04:59:39.056685 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:39.056699 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:39.056771 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:39.056884 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:39.056885 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:39.057044 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:39.057092 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:39.057093 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:39.057773 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:39.057774 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:39.058452 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:39.058456 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:39.058831 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:39.058832 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:39.058873 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:39.059289 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:39.059292 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:39.061066 (+ 1774us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":69,"threads_started":1}
W20260924 04:59:39.061194 9574 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:59:39.056719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55677 (local address 127.0.98.62:37517)
0924 04:59:39.056779 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:39.056780 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:39.056789 (+ 9us) server_negotiation.cc:407] Connection header received
0924 04:59:39.056816 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:39.056816 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:39.056826 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:39.056873 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:39.057144 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:39.057703 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:39.058529 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:39.058773 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:39.058824 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:39.061146 (+ 2322us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:55677: BlockingRecv error: recv got EOF from 127.0.98.3:55677 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 24 04:59:39 dist-test-slave-70st krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:59:39 dist-test-slave-70st krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:59:39.837023 9577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:39.826131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.1:34247)
0924 04:59:39.826251 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:39.826261 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:39.826309 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:39.826447 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:39.826447 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:39.826602 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:39.826646 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:39.826648 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:39.827275 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:39.827276 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:39.827784 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:39.827785 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:39.828078 (+ 293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:39.828078 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:39.828113 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:39.835358 (+ 7245us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:39.835364 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:39.836960 (+ 1596us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":56,"threads_started":1}
W20260924 04:59:39.837138 9578 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:59:39.826190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34247 (local address 127.0.98.62:37517)
0924 04:59:39.826299 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:39.826300 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:39.826309 (+ 9us) server_negotiation.cc:407] Connection header received
0924 04:59:39.826355 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:39.826355 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:39.826365 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:39.826439 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:39.826699 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:39.827219 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:39.827840 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:39.828031 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:39.828074 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:39.837069 (+ 8995us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:34247: BlockingRecv error: recv got EOF from 127.0.98.1:34247 (error 108)
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":51,"threads_started":1}
Sep 24 04:59:39 dist-test-slave-70st krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:59:39 dist-test-slave-70st krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:59:39.944830 9579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:39.940116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.2:54723)
0924 04:59:39.940223 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:39.940235 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:39.940294 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:39.940427 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:39.940428 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:39.940577 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:39.940625 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:39.940626 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:39.941347 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:39.941348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:39.941927 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:39.941930 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:39.942239 (+ 309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:39.942240 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:39.942276 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:39.943182 (+ 906us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:39.943186 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:39.944770 (+ 1584us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":63,"threads_started":1}
W20260924 04:59:39.944952 9578 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:59:39.940201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54723 (local address 127.0.98.62:37517)
0924 04:59:39.940257 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:39.940259 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:39.940329 (+ 70us) server_negotiation.cc:407] Connection header received
0924 04:59:39.940358 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:39.940359 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:39.940371 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:39.940419 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:39.940711 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:39.941260 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:39.941993 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:39.942185 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:39.942236 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:39.944884 (+ 2648us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:54723: BlockingRecv error: recv got EOF from 127.0.98.2:54723 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 24 04:59:40 dist-test-slave-70st krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:59:40 dist-test-slave-70st krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:59:40.066354 9580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:40.061600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.3:47389)
0924 04:59:40.061717 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:40.061729 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:40.061799 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:40.061961 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:40.061962 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:40.062117 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:40.062164 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:40.062166 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:40.062806 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:40.062807 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:40.063362 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:40.063364 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:40.063650 (+ 286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:40.063650 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:40.063685 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:40.064626 (+ 941us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:40.064630 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:40.066293 (+ 1663us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":54,"threads_started":1}
W20260924 04:59:40.066449 9578 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:59:40.061696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47389 (local address 127.0.98.62:37517)
0924 04:59:40.061744 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:40.061745 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:40.061783 (+ 38us) server_negotiation.cc:407] Connection header received
0924 04:59:40.061843 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:40.061844 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:40.061856 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:40.061904 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:40.062230 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:40.062740 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:40.063428 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:40.063600 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:40.063645 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:40.066390 (+ 2745us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:47389: BlockingRecv error: recv got EOF from 127.0.98.3:47389 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Sep 24 04:59:40 dist-test-slave-70st krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:59:40 dist-test-slave-70st krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:59:40.847926 9581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:40.837503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.1:52425)
0924 04:59:40.837640 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:40.837651 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:40.837718 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:40.837848 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:40.837849 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:40.838018 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:40.838083 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:40.838085 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:40.838771 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:40.838772 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:40.839291 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:40.839292 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:40.839648 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:40.839649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:40.839733 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:40.845994 (+ 6261us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:40.845999 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:40.847841 (+ 1842us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":70,"threads_started":1}
W20260924 04:59:40.848037 9582 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:59:40.837566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52425 (local address 127.0.98.62:37517)
0924 04:59:40.837659 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:40.837662 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:40.837702 (+ 40us) server_negotiation.cc:407] Connection header received
0924 04:59:40.837772 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:40.837773 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:40.837782 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:40.837841 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:40.838134 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:40.838714 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:40.839371 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:40.839564 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:40.839615 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:40.847960 (+ 8345us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:52425: BlockingRecv error: recv got EOF from 127.0.98.1:52425 (error 108)
Metrics: {"server-negotiator.queue_time_us":73,"thread_start_us":48,"threads_started":1}
Sep 24 04:59:40 dist-test-slave-70st krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:59:40 dist-test-slave-70st krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:59:40.949446 9583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:40.945216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.2:56973)
0924 04:59:40.945326 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:40.945336 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:40.945400 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:40.945564 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:40.945565 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:40.945714 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:40.945764 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:40.945766 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:40.946395 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:40.946395 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:40.946926 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:40.946927 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:40.947211 (+ 284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:40.947211 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:40.947248 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:40.947678 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:40.947682 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:40.949326 (+ 1644us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":54,"threads_started":1}
W20260924 04:59:40.949501 9582 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:59:40.945262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56973 (local address 127.0.98.62:37517)
0924 04:59:40.945335 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:40.945337 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:40.945379 (+ 42us) server_negotiation.cc:407] Connection header received
0924 04:59:40.945500 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:40.945500 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:40.945511 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:40.945557 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:40.945825 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:40.946324 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:40.946982 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:40.947163 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:40.947206 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:40.949448 (+ 2242us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:56973: BlockingRecv error: recv got EOF from 127.0.98.2:56973 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 24 04:59:41 dist-test-slave-70st krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:59:41 dist-test-slave-70st krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:59:41.071264 9584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:41.066856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.3:47635)
0924 04:59:41.067035 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:41.067045 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:41.067105 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:41.067237 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:41.067238 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:41.067409 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:41.067454 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:41.067456 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:41.068086 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:41.068087 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:41.068611 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:41.068613 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:41.068944 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:41.068945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:41.069003 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:41.069456 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:41.069460 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:41.071190 (+ 1730us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":57,"threads_started":1}
W20260924 04:59:41.071364 9582 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:59:41.066940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47635 (local address 127.0.98.62:37517)
0924 04:59:41.066991 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:41.066993 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:41.067105 (+ 112us) server_negotiation.cc:407] Connection header received
0924 04:59:41.067163 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:41.067164 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:41.067178 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:41.067222 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:41.067527 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:41.068012 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:41.068690 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:41.068881 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:41.068933 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:41.071303 (+ 2370us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:47635: BlockingRecv error: recv got EOF from 127.0.98.3:47635 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Sep 24 04:59:41 dist-test-slave-70st krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:59:41 dist-test-slave-70st krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:59:41.852834 9585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:41.848377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.1:52255)
0924 04:59:41.848522 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:41.848535 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:41.848610 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:41.848770 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:41.848771 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:41.848923 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:41.848973 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:41.848974 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:41.849765 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:41.849766 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:41.850298 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:41.850300 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:41.850608 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:41.850609 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:41.850657 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:41.851102 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:41.851107 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:41.852760 (+ 1653us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":67,"threads_started":1}
W20260924 04:59:41.852934 9586 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:59:41.848460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52255 (local address 127.0.98.62:37517)
0924 04:59:41.848591 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:41.848593 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:41.848601 (+ 8us) server_negotiation.cc:407] Connection header received
0924 04:59:41.848664 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:41.848665 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:41.848675 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:41.848731 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:41.849037 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:41.849702 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:41.850353 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:41.850536 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:41.850579 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:41.852846 (+ 2267us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:52255: BlockingRecv error: recv got EOF from 127.0.98.1:52255 (error 108)
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":73,"threads_started":1}
Sep 24 04:59:41 dist-test-slave-70st krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:59:41 dist-test-slave-70st krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:59:41.954255 9587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:41.949928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.2:44067)
0924 04:59:41.950085 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:41.950096 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:41.950151 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:41.950273 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:41.950274 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:41.950425 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:41.950472 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:41.950473 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:41.951139 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:41.951140 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:41.951679 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:41.951681 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:41.952037 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:41.952038 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:41.952090 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:41.952514 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:41.952518 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:41.954197 (+ 1679us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":73,"threads_started":1}
W20260924 04:59:41.954353 9586 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:59:41.950084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44067 (local address 127.0.98.62:37517)
0924 04:59:41.950142 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:41.950144 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:41.950153 (+ 9us) server_negotiation.cc:407] Connection header received
0924 04:59:41.950198 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:41.950198 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:41.950208 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:41.950265 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:41.950551 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:41.951060 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:41.951753 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:41.951954 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:41.952010 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:41.954289 (+ 2279us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:44067: BlockingRecv error: recv got EOF from 127.0.98.2:44067 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 24 04:59:42 dist-test-slave-70st krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:59:42 dist-test-slave-70st krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:59:42.076175 9588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:42.071770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.3:59095)
0924 04:59:42.071896 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:42.071907 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:42.071963 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:42.072077 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:42.072078 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:42.072231 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:42.072278 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:42.072279 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:42.072982 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:42.072984 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:42.073538 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:42.073540 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:42.073838 (+ 298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:42.073839 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:42.073874 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:42.074319 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:42.074323 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:42.076115 (+ 1792us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":62,"threads_started":1}
W20260924 04:59:42.076294 9586 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:59:42.071850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59095 (local address 127.0.98.62:37517)
0924 04:59:42.071901 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:42.071903 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:42.071949 (+ 46us) server_negotiation.cc:407] Connection header received
0924 04:59:42.072017 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:42.072017 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:42.072029 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:42.072065 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:42.072326 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:42.072920 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:42.073606 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:42.073784 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:42.073833 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:42.076203 (+ 2370us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:59095: BlockingRecv error: recv got EOF from 127.0.98.3:59095 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Sep 24 04:59:42 dist-test-slave-70st krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:59:42 dist-test-slave-70st krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:59:42.857861 9589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:42.853313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.1:35533)
0924 04:59:42.853470 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:42.853480 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:42.853526 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:42.853644 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:42.853645 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:42.853799 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:42.853854 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:42.853855 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:42.854471 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:42.854472 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:42.855003 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:42.855005 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:42.855409 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:42.855410 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:42.855460 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:42.855972 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:42.855976 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:42.857785 (+ 1809us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":88,"threads_started":1}
W20260924 04:59:42.857942 9590 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:59:42.853382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35533 (local address 127.0.98.62:37517)
0924 04:59:42.853523 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:42.853525 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:42.853535 (+ 10us) server_negotiation.cc:407] Connection header received
0924 04:59:42.853576 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:42.853576 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:42.853586 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:42.853632 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:42.853912 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:42.854411 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:42.855099 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:42.855321 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:42.855392 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:42.857881 (+ 2489us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:35533: BlockingRecv error: recv got EOF from 127.0.98.1:35533 (error 108)
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":80,"threads_started":1}
Sep 24 04:59:42 dist-test-slave-70st krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:59:42 dist-test-slave-70st krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:59:42.959013 9591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:42.954700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.2:54097)
0924 04:59:42.954831 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:42.954842 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:42.954906 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:42.955032 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:42.955033 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:42.955182 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:42.955228 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:42.955229 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:42.955823 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:42.955824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:42.956373 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:42.956375 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:42.956656 (+ 281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:42.956657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:42.956692 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:42.957117 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:42.957121 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:42.958947 (+ 1826us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":78,"threads_started":1}
W20260924 04:59:42.959085 9590 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:59:42.954756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54097 (local address 127.0.98.62:37517)
0924 04:59:42.954819 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:42.954821 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:42.954900 (+ 79us) server_negotiation.cc:407] Connection header received
0924 04:59:42.954961 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:42.954962 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:42.954972 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:42.955023 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:42.955276 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:42.955761 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:42.956434 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:42.956610 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:42.956653 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:42.959031 (+ 2378us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:54097: BlockingRecv error: recv got EOF from 127.0.98.2:54097 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 24 04:59:43 dist-test-slave-70st krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:59:43 dist-test-slave-70st krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:59:43.080874 9592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:43.076657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.3:52535)
0924 04:59:43.076801 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:43.076812 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:43.076877 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:43.076996 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:43.076997 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:43.077151 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:43.077197 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:43.077198 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:43.077965 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:43.077966 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:43.078507 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:43.078509 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:43.078794 (+ 285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:43.078795 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:43.078833 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:43.079223 (+ 390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:43.079227 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:43.080816 (+ 1589us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":79,"threads_started":1}
W20260924 04:59:43.080955 9590 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:59:43.076746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52535 (local address 127.0.98.62:37517)
0924 04:59:43.076806 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:43.076808 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:43.076875 (+ 67us) server_negotiation.cc:407] Connection header received
0924 04:59:43.076922 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:43.076923 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:43.076934 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:43.076984 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:43.077268 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:43.077896 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:43.078567 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:43.078746 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:43.078790 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:43.080897 (+ 2107us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:52535: BlockingRecv error: recv got EOF from 127.0.98.3:52535 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 24 04:59:43 dist-test-slave-70st krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:59:43 dist-test-slave-70st krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:59:43.862661 9593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:43.858362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.1:56113)
0924 04:59:43.858506 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:43.858515 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:43.858563 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:43.858697 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:43.858697 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:43.858848 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:43.858903 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:43.858905 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:43.859611 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:43.859611 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:43.860131 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:43.860132 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:43.860504 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:43.860504 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:43.860539 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:43.860976 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:43.860980 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:43.862594 (+ 1614us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":87,"threads_started":1}
W20260924 04:59:43.862757 9594 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:59:43.858431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56113 (local address 127.0.98.62:37517)
0924 04:59:43.858553 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:43.858555 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:43.858564 (+ 9us) server_negotiation.cc:407] Connection header received
0924 04:59:43.858617 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:43.858617 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:43.858627 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:43.858689 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:43.858996 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:43.859546 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:43.860240 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:43.860441 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:43.860503 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:43.862692 (+ 2189us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:56113: BlockingRecv error: recv got EOF from 127.0.98.1:56113 (error 108)
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":55,"threads_started":1}
Sep 24 04:59:43 dist-test-slave-70st krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 04:59:43 dist-test-slave-70st krb5kdc[8536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 04:59:43.964249 9595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:43.959576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37517 (local address 127.0.98.2:33263)
0924 04:59:43.959710 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:43.959720 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:43.959785 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:43.959983 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:43.959984 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:43.960139 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:43.960196 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:43.960198 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:43.961090 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:43.961091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:43.961679 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:43.961681 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:43.961976 (+ 295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:43.961976 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:43.962013 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:43.962443 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:43.962447 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:43.964179 (+ 1732us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37517: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":79,"threads_started":1}
W20260924 04:59:43.964308 9594 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:59:43.959662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33263 (local address 127.0.98.62:37517)
0924 04:59:43.959732 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:43.959734 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:43.959790 (+ 56us) server_negotiation.cc:407] Connection header received
0924 04:59:43.959834 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:43.959834 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:43.959844 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:43.959906 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:43.960276 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:43.961021 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:43.961737 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:43.961922 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:43.961969 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:43.964260 (+ 2291us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:33263: BlockingRecv error: recv got EOF from 127.0.98.2:33263 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 04:59:44.006691 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 9119
I20260924 04:59:44.010731 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 9269
I20260924 04:59:44.014413 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 9404
I20260924 04:59:44.017891 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 9049
2026-09-24T04:59:44Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17253 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-24T04:59:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:59:44Z Disabled control of system clock
I20260924 04:59:44.043265 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225929848825-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225929848825-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225929848825-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33119
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:44569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33119
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 04:59:44.122808 9599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:59:44.123044 9599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:59:44.123072 9599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:59:44.124272 9599 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:59:44.124318 9599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:59:44.124334 9599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:59:44.124351 9599 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:59:44.124365 9599 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:59:44.125771 9599 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225929848825-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225929848825-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33119
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33119
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9599
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225929848825-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 04:59:44.126035 9599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:59:44.126314 9599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:59:44.131335 9604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:59:44.131402 9605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:59:44.131415 9599 server_base.cc:1034] running on GCE node
W20260924 04:59:44.131456 9607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:59:44.131829 9599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:59:44.132119 9599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:59:44.133275 9599 hybrid_clock.cc:648] HybridClock initialized: now 1790225984133258 us; error 33 us; skew 500 ppm
I20260924 04:59:44.134591 9599 webserver.cc:460] Webserver started at http://127.0.98.62:45349/ using document root <none> and password file <none>
I20260924 04:59:44.134819 9599 fs_manager.cc:359] Metadata directory not provided
I20260924 04:59:44.134881 9599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:59:44.135006 9599 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:59:44.136145 9599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225929848825-392-0/minicluster-data/master-0/data/instance:
uuid: "02043612a1b045c19ad5d77172b36b14"
format_stamp: "Formatted at 2026-09-24 04:59:44 on dist-test-slave-70st"
server_key: "bbb0108dc4f6232bea29f3873db317d3"
server_key_iv: "a2d3ad2a8577e45bc1a2196ea16c60dc"
server_key_version: "encryptionkey@0"
I20260924 04:59:44.136512 9599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225929848825-392-0/minicluster-data/master-0/wal/instance:
uuid: "02043612a1b045c19ad5d77172b36b14"
format_stamp: "Formatted at 2026-09-24 04:59:44 on dist-test-slave-70st"
server_key: "bbb0108dc4f6232bea29f3873db317d3"
server_key_iv: "a2d3ad2a8577e45bc1a2196ea16c60dc"
server_key_version: "encryptionkey@0"
I20260924 04:59:44.137895 9599 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 04:59:44.138612 9613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:44.138763 9599 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 04:59:44.138849 9599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225929848825-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225929848825-392-0/minicluster-data/master-0/wal
uuid: "02043612a1b045c19ad5d77172b36b14"
format_stamp: "Formatted at 2026-09-24 04:59:44 on dist-test-slave-70st"
server_key: "bbb0108dc4f6232bea29f3873db317d3"
server_key_iv: "a2d3ad2a8577e45bc1a2196ea16c60dc"
server_key_version: "encryptionkey@0"
I20260924 04:59:44.138914 9599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225929848825-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225929848825-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225929848825-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:59:44.148260 9599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:59:44.148533 9599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:59:44.148662 9599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:59:44.152509 9599 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33119
I20260924 04:59:44.152544 9665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33119 every 8 connection(s)
I20260924 04:59:44.152866 9599 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:59:44.153545 9666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:59:44.156059 9666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02043612a1b045c19ad5d77172b36b14: Bootstrap starting.
I20260924 04:59:44.156620 9666 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 02043612a1b045c19ad5d77172b36b14: Neither blocks nor log segments found. Creating new log.
I20260924 04:59:44.156874 9666 log.cc:824] T 00000000000000000000000000000000 P 02043612a1b045c19ad5d77172b36b14: Log is configured to *not* fsync() on all Append() calls
I20260924 04:59:44.157552 9666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02043612a1b045c19ad5d77172b36b14: No bootstrap required, opened a new log
I20260924 04:59:44.157594 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 9599
I20260924 04:59:44.157671 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225929848825-392-0/minicluster-data/master-0/wal/instance
I20260924 04:59:44.157840 392 external_mini_cluster.cc:1428] Setting key 919a3aa7eedc0901c003d9ad17993df9
I20260924 04:59:44.159514 9666 raft_consensus.cc:348] T 00000000000000000000000000000000 P 02043612a1b045c19ad5d77172b36b14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02043612a1b045c19ad5d77172b36b14" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33119 } }
I20260924 04:59:44.159686 9666 raft_consensus.cc:374] T 00000000000000000000000000000000 P 02043612a1b045c19ad5d77172b36b14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:59:44.159720 9666 raft_consensus.cc:729] T 00000000000000000000000000000000 P 02043612a1b045c19ad5d77172b36b14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02043612a1b045c19ad5d77172b36b14, State: Initialized, Role: FOLLOWER
I20260924 04:59:44.159873 9666 consensus_queue.cc:260] T 00000000000000000000000000000000 P 02043612a1b045c19ad5d77172b36b14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02043612a1b045c19ad5d77172b36b14" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33119 } }
I20260924 04:59:44.159968 9666 raft_consensus.cc:388] T 00000000000000000000000000000000 P 02043612a1b045c19ad5d77172b36b14 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:59:44.159998 9666 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02043612a1b045c19ad5d77172b36b14 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:59:44.160045 9666 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 02043612a1b045c19ad5d77172b36b14 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:59:44.160709 9666 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02043612a1b045c19ad5d77172b36b14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02043612a1b045c19ad5d77172b36b14" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33119 } }
I20260924 04:59:44.160815 9666 leader_election.cc:302] T 00000000000000000000000000000000 P 02043612a1b045c19ad5d77172b36b14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 02043612a1b045c19ad5d77172b36b14; no voters:
I20260924 04:59:44.160975 9666 leader_election.cc:288] T 00000000000000000000000000000000 P 02043612a1b045c19ad5d77172b36b14 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:59:44.161016 9671 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 02043612a1b045c19ad5d77172b36b14 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:59:44.161168 9671 raft_consensus.cc:686] T 00000000000000000000000000000000 P 02043612a1b045c19ad5d77172b36b14 [term 1 LEADER]: Becoming Leader. State: Replica: 02043612a1b045c19ad5d77172b36b14, State: Running, Role: LEADER
I20260924 04:59:44.161201 9666 sys_catalog.cc:564] T 00000000000000000000000000000000 P 02043612a1b045c19ad5d77172b36b14 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:59:44.161314 9671 consensus_queue.cc:237] T 00000000000000000000000000000000 P 02043612a1b045c19ad5d77172b36b14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02043612a1b045c19ad5d77172b36b14" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33119 } }
I20260924 04:59:44.161679 9673 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02043612a1b045c19ad5d77172b36b14 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "02043612a1b045c19ad5d77172b36b14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02043612a1b045c19ad5d77172b36b14" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33119 } } }
I20260924 04:59:44.161824 9673 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02043612a1b045c19ad5d77172b36b14 [sys.catalog]: This master's current role is: LEADER
I20260924 04:59:44.162524 9672 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02043612a1b045c19ad5d77172b36b14 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 02043612a1b045c19ad5d77172b36b14. Latest consensus state: current_term: 1 leader_uuid: "02043612a1b045c19ad5d77172b36b14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02043612a1b045c19ad5d77172b36b14" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33119 } } }
I20260924 04:59:44.162879 9672 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02043612a1b045c19ad5d77172b36b14 [sys.catalog]: This master's current role is: LEADER
W20260924 04:59:44.163089 9686 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 02043612a1b045c19ad5d77172b36b14: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 04:59:44.163132 9686 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 04:59:44.163677 9680 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:59:44.163923 9680 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:59:44.164613 9670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:44.158349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45890 (local address 127.0.98.62:33119)
0924 04:59:44.158559 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:44.158568 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:44.158601 (+ 33us) server_negotiation.cc:407] Connection header received
0924 04:59:44.159479 (+ 878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:44.159488 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:44.159572 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:44.159727 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:44.159916 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:44.161594 (+ 1678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:44.162674 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:44.163268 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:44.163378 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:44.163984 (+ 606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:44.163988 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:44.163988 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:59:44.164157 (+ 169us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:44.164189 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:59:44.164199 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:44.164255 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:44.164339 (+ 84us) server_negotiation.cc:299] Negotiation successful
0924 04:59:44.164416 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":84,"threads_started":1}
I20260924 04:59:44.165608 9680 catalog_manager.cc:1347] Generated new cluster ID: 2aecb6b1403b48eeb0b580c669af99f1
I20260924 04:59:44.165649 9680 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:59:44.185716 9680 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:59:44.186241 9680 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:59:44.190912 9680 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 02043612a1b045c19ad5d77172b36b14: Generated new TSK 0
I20260924 04:59:44.191082 9680 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:59:44.196996 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225929848825-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225929848825-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225929848825-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225929848825-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33119
--builtin_ntp_servers=127.0.98.20:44569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260924 04:59:44.281597 9690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:59:44.281796 9690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:59:44.281824 9690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:59:44.282963 9690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:59:44.283001 9690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:59:44.283034 9690 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:59:44.284381 9690 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225929848825-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225929848825-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225929848825-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33119
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9690
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225929848825-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 04:59:44.284600 9690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:59:44.284859 9690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:59:44.290380 9696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:59:44.290366 9695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:59:44.290428 9690 server_base.cc:1034] running on GCE node
W20260924 04:59:44.290653 9698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:59:44.290856 9690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:59:44.291090 9690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:59:44.292228 9690 hybrid_clock.cc:648] HybridClock initialized: now 1790225984292214 us; error 32 us; skew 500 ppm
I20260924 04:59:44.293409 9690 webserver.cc:460] Webserver started at http://127.0.98.1:33421/ using document root <none> and password file <none>
I20260924 04:59:44.293650 9690 fs_manager.cc:359] Metadata directory not provided
I20260924 04:59:44.293697 9690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:59:44.293821 9690 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:59:44.294842 9690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225929848825-392-0/minicluster-data/ts-0/data/instance:
uuid: "01110048785d45a5ba513b857e134f46"
format_stamp: "Formatted at 2026-09-24 04:59:44 on dist-test-slave-70st"
server_key: "742ec1a4ef528b75c82fd4b69caf62f0"
server_key_iv: "d726064eaeb40b6671e3674bb4b4bb85"
server_key_version: "encryptionkey@0"
I20260924 04:59:44.295181 9690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225929848825-392-0/minicluster-data/ts-0/wal/instance:
uuid: "01110048785d45a5ba513b857e134f46"
format_stamp: "Formatted at 2026-09-24 04:59:44 on dist-test-slave-70st"
server_key: "742ec1a4ef528b75c82fd4b69caf62f0"
server_key_iv: "d726064eaeb40b6671e3674bb4b4bb85"
server_key_version: "encryptionkey@0"
I20260924 04:59:44.296404 9690 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 04:59:44.297093 9704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:44.297250 9690 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 04:59:44.297327 9690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225929848825-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225929848825-392-0/minicluster-data/ts-0/wal
uuid: "01110048785d45a5ba513b857e134f46"
format_stamp: "Formatted at 2026-09-24 04:59:44 on dist-test-slave-70st"
server_key: "742ec1a4ef528b75c82fd4b69caf62f0"
server_key_iv: "d726064eaeb40b6671e3674bb4b4bb85"
server_key_version: "encryptionkey@0"
I20260924 04:59:44.297434 9690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225929848825-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225929848825-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225929848825-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:59:44.308651 9690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:59:44.308933 9690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:59:44.309052 9690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:59:44.309239 9690 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:59:44.309580 9690 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:59:44.309614 9690 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:44.309641 9690 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:59:44.309666 9690 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:44.314847 9690 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45103
I20260924 04:59:44.314913 9817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45103 every 8 connection(s)
I20260924 04:59:44.315207 9690 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225929848825-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:59:44.319844 9670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:44.315858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39793 (local address 127.0.98.62:33119)
0924 04:59:44.315920 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:44.315923 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:44.316157 (+ 234us) server_negotiation.cc:407] Connection header received
0924 04:59:44.316631 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:44.316631 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:44.316642 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:44.316699 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:44.317277 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:44.317811 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:44.318701 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:44.318892 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:44.318937 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:44.319669 (+ 732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:44.319669 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:44.319669 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:59:44.319673 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:44.319682 (+ 9us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:59:44.319686 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:44.319715 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:44.319770 (+ 55us) server_negotiation.cc:299] Negotiation successful
0924 04:59:44.319787 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 04:59:44.320027 9820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:44.315833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33119 (local address 127.0.98.1:39793)
0924 04:59:44.316049 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:44.316082 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:44.316545 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:44.316754 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:44.316757 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:44.316765 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:44.317217 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:44.317221 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:44.317878 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:44.317878 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:44.318638 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:44.318640 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:44.318948 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:44.318949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:44.319049 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:44.319557 (+ 508us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:59:44.319558 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:44.319586 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:59:44.319608 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:59:44.319717 (+ 109us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:59:44.319720 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:59:44.319723 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 04:59:44.319766 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 04:59:44.319848 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":75,"threads_started":1}
I20260924 04:59:44.320494 9818 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33119
I20260924 04:59:44.320566 9818 heartbeater.cc:464] Registering TS with master...
I20260924 04:59:44.320706 9818 heartbeater.cc:511] Master 127.0.98.62:33119 requested a full tablet report, sending...
I20260924 04:59:44.321146 9630 ts_manager.cc:194] Registered new tserver with Master: 01110048785d45a5ba513b857e134f46 (127.0.98.1:45103)
I20260924 04:59:44.321799 9630 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:39793
I20260924 04:59:44.322077 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 9690
I20260924 04:59:44.322160 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225929848825-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:59:44.322328 392 external_mini_cluster.cc:1428] Setting key 5e04eb8ec578a15fe205fe9cb68548da
I20260924 04:59:44.323074 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225929848825-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225929848825-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225929848825-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225929848825-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33119
--builtin_ntp_servers=127.0.98.20:44569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260924 04:59:44.400638 9821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:59:44.400856 9821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:59:44.400890 9821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:59:44.402086 9821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:59:44.402125 9821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:59:44.402158 9821 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:59:44.403434 9821 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225929848825-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225929848825-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225929848825-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33119
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9821
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225929848825-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 04:59:44.403654 9821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:59:44.403920 9821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:59:44.408612 9826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:59:44.408674 9821 server_base.cc:1034] running on GCE node
W20260924 04:59:44.408601 9827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:59:44.408601 9829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:59:44.408990 9821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:59:44.409209 9821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:59:44.410343 9821 hybrid_clock.cc:648] HybridClock initialized: now 1790225984410321 us; error 38 us; skew 500 ppm
I20260924 04:59:44.411465 9821 webserver.cc:460] Webserver started at http://127.0.98.2:45011/ using document root <none> and password file <none>
I20260924 04:59:44.411667 9821 fs_manager.cc:359] Metadata directory not provided
I20260924 04:59:44.411710 9821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:59:44.411824 9821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:59:44.412854 9821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225929848825-392-0/minicluster-data/ts-1/data/instance:
uuid: "bbc10516f15048a6ae2e9bff625bb5c5"
format_stamp: "Formatted at 2026-09-24 04:59:44 on dist-test-slave-70st"
server_key: "ee9d51d2fdbb3705a8f4a66574c189f5"
server_key_iv: "ea83061e8d2f0cb953e53ea7c7aa7eb4"
server_key_version: "encryptionkey@0"
I20260924 04:59:44.413182 9821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225929848825-392-0/minicluster-data/ts-1/wal/instance:
uuid: "bbc10516f15048a6ae2e9bff625bb5c5"
format_stamp: "Formatted at 2026-09-24 04:59:44 on dist-test-slave-70st"
server_key: "ee9d51d2fdbb3705a8f4a66574c189f5"
server_key_iv: "ea83061e8d2f0cb953e53ea7c7aa7eb4"
server_key_version: "encryptionkey@0"
I20260924 04:59:44.414481 9821 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 04:59:44.415216 9835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:44.415345 9821 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 04:59:44.415424 9821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225929848825-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225929848825-392-0/minicluster-data/ts-1/wal
uuid: "bbc10516f15048a6ae2e9bff625bb5c5"
format_stamp: "Formatted at 2026-09-24 04:59:44 on dist-test-slave-70st"
server_key: "ee9d51d2fdbb3705a8f4a66574c189f5"
server_key_iv: "ea83061e8d2f0cb953e53ea7c7aa7eb4"
server_key_version: "encryptionkey@0"
I20260924 04:59:44.415490 9821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225929848825-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225929848825-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225929848825-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:59:44.432438 9821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:59:44.432727 9821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:59:44.432854 9821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:59:44.433079 9821 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:59:44.433394 9821 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:59:44.433473 9821 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:44.433506 9821 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:59:44.433529 9821 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:44.438915 9821 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38025
I20260924 04:59:44.438952 9948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38025 every 8 connection(s)
I20260924 04:59:44.439261 9821 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225929848825-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:59:44.444271 9670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:44.439935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45853 (local address 127.0.98.62:33119)
0924 04:59:44.440017 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:44.440019 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:44.440201 (+ 182us) server_negotiation.cc:407] Connection header received
0924 04:59:44.440721 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:44.440723 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:44.440733 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:44.440782 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:44.441492 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:44.441993 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:44.443079 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:44.443259 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:44.443309 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:44.444082 (+ 773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:44.444083 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:44.444084 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:59:44.444087 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:44.444097 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:59:44.444100 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:44.444132 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:44.444195 (+ 63us) server_negotiation.cc:299] Negotiation successful
0924 04:59:44.444214 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 04:59:44.444489 9951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:44.439916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33119 (local address 127.0.98.2:45853)
0924 04:59:44.440088 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:44.440122 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:44.440626 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:44.440848 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:44.440851 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:44.440860 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:44.441385 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:44.441391 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:44.442094 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:44.442094 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:44.443013 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:44.443016 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:44.443330 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:44.443331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:44.443442 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:44.443966 (+ 524us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:59:44.443966 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:44.443996 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:59:44.444020 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:59:44.444141 (+ 121us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:59:44.444144 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:59:44.444146 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 04:59:44.444193 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 04:59:44.444292 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":77,"threads_started":1}
I20260924 04:59:44.444962 9949 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33119
I20260924 04:59:44.445065 9949 heartbeater.cc:464] Registering TS with master...
I20260924 04:59:44.445240 9949 heartbeater.cc:511] Master 127.0.98.62:33119 requested a full tablet report, sending...
I20260924 04:59:44.445650 9630 ts_manager.cc:194] Registered new tserver with Master: bbc10516f15048a6ae2e9bff625bb5c5 (127.0.98.2:38025)
I20260924 04:59:44.445990 9630 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:45853
I20260924 04:59:44.447434 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 9821
I20260924 04:59:44.447515 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225929848825-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:59:44.447682 392 external_mini_cluster.cc:1428] Setting key c4b77bf8d7911d2f82de8c4f5eeba3df
I20260924 04:59:44.448513 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225929848825-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225929848825-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225929848825-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225929848825-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33119
--builtin_ntp_servers=127.0.98.20:44569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260924 04:59:44.527338 9952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:59:44.527550 9952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:59:44.527570 9952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:59:44.528731 9952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:59:44.528771 9952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:59:44.528797 9952 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:59:44.530134 9952 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225929848825-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225929848825-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225929848825-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33119
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9952
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225929848825-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 04:59:44.530328 9952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:59:44.530561 9952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:59:44.535194 9958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:59:44.535277 9960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:59:44.535188 9952 server_base.cc:1034] running on GCE node
W20260924 04:59:44.535169 9957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:59:44.535565 9952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:59:44.535799 9952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:59:44.536931 9952 hybrid_clock.cc:648] HybridClock initialized: now 1790225984536914 us; error 29 us; skew 500 ppm
I20260924 04:59:44.538115 9952 webserver.cc:460] Webserver started at http://127.0.98.3:46621/ using document root <none> and password file <none>
I20260924 04:59:44.538316 9952 fs_manager.cc:359] Metadata directory not provided
I20260924 04:59:44.538362 9952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:59:44.538517 9952 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:59:44.539541 9952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225929848825-392-0/minicluster-data/ts-2/data/instance:
uuid: "8ebbc31f77754ea0818eefb1887ffdff"
format_stamp: "Formatted at 2026-09-24 04:59:44 on dist-test-slave-70st"
server_key: "2439033ad653c1ce41e3f74d51a68d9b"
server_key_iv: "8e7f01e29e1b6be803675a03b6329bfb"
server_key_version: "encryptionkey@0"
I20260924 04:59:44.539860 9952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225929848825-392-0/minicluster-data/ts-2/wal/instance:
uuid: "8ebbc31f77754ea0818eefb1887ffdff"
format_stamp: "Formatted at 2026-09-24 04:59:44 on dist-test-slave-70st"
server_key: "2439033ad653c1ce41e3f74d51a68d9b"
server_key_iv: "8e7f01e29e1b6be803675a03b6329bfb"
server_key_version: "encryptionkey@0"
I20260924 04:59:44.541056 9952 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 04:59:44.541733 9966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:44.541878 9952 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:44.541957 9952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225929848825-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225929848825-392-0/minicluster-data/ts-2/wal
uuid: "8ebbc31f77754ea0818eefb1887ffdff"
format_stamp: "Formatted at 2026-09-24 04:59:44 on dist-test-slave-70st"
server_key: "2439033ad653c1ce41e3f74d51a68d9b"
server_key_iv: "8e7f01e29e1b6be803675a03b6329bfb"
server_key_version: "encryptionkey@0"
I20260924 04:59:44.542021 9952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225929848825-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225929848825-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225929848825-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:59:44.560086 9952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:59:44.560418 9952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:59:44.560547 9952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:59:44.560752 9952 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:59:44.561075 9952 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:59:44.561110 9952 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:44.561137 9952 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:59:44.561161 9952 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:44.566546 9952 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46743
I20260924 04:59:44.566610 10079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46743 every 8 connection(s)
I20260924 04:59:44.566901 9952 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225929848825-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:59:44.571636 9670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:44.567574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52917 (local address 127.0.98.62:33119)
0924 04:59:44.567636 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:44.567638 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:44.567863 (+ 225us) server_negotiation.cc:407] Connection header received
0924 04:59:44.568332 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:44.568333 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:44.568342 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:44.568402 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:44.569003 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:44.569556 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:44.570485 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:44.570660 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:44.570705 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:44.571455 (+ 750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:44.571456 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:44.571456 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:59:44.571460 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:44.571469 (+ 9us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:59:44.571471 (+ 2us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:44.571502 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:44.571558 (+ 56us) server_negotiation.cc:299] Negotiation successful
0924 04:59:44.571578 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 04:59:44.571837 10082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:44.567548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33119 (local address 127.0.98.3:52917)
0924 04:59:44.567738 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:44.567780 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:44.568255 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:44.568457 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:44.568461 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:44.568469 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:44.568938 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:44.568942 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:44.569632 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:44.569632 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:44.570423 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:44.570425 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:44.570713 (+ 288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:44.570713 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:44.570811 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:44.571314 (+ 503us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:59:44.571314 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:44.571370 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:59:44.571393 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:59:44.571505 (+ 112us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:59:44.571507 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:59:44.571510 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 04:59:44.571554 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 04:59:44.571630 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":83,"threads_started":1}
I20260924 04:59:44.572281 10080 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33119
I20260924 04:59:44.572386 10080 heartbeater.cc:464] Registering TS with master...
I20260924 04:59:44.572554 10080 heartbeater.cc:511] Master 127.0.98.62:33119 requested a full tablet report, sending...
I20260924 04:59:44.572901 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 9952
I20260924 04:59:44.572903 9630 ts_manager.cc:194] Registered new tserver with Master: 8ebbc31f77754ea0818eefb1887ffdff (127.0.98.3:46743)
I20260924 04:59:44.572978 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790225929848825-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:59:44.573146 392 external_mini_cluster.cc:1428] Setting key 0e132910fc79ebe46bc9dd677b8ca7b1
I20260924 04:59:44.573283 9630 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:52917
I20260924 04:59:44.573944 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 04:59:44.575793 9670 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:59:44.575236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45900 (local address 127.0.98.62:33119)
0924 04:59:44.575279 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:44.575281 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:44.575426 (+ 145us) server_negotiation.cc:407] Connection header received
0924 04:59:44.575503 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:44.575503 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:44.575515 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:44.575589 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:44.575729 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45900: BlockingRecv error: recv got EOF from 127.0.0.1:45900 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 04:59:44.575984 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 9690
I20260924 04:59:44.580652 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 9821
I20260924 04:59:44.584110 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 9952
I20260924 04:59:44.587644 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 9599
2026-09-24T04:59:44Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (568 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-24T04:59:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:59:44Z Disabled control of system clock
I20260924 04:59:44.611704 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225929848825-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225929848825-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225929848825-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40609
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:46135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40609
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 04:59:44.688565 10091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:59:44.688764 10091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:59:44.688784 10091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:59:44.689957 10091 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:59:44.689998 10091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:59:44.690011 10091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:59:44.690021 10091 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:59:44.690032 10091 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:59:44.691341 10091 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225929848825-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225929848825-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40609
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40609
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10091
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225929848825-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 04:59:44.691572 10091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:59:44.691848 10091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:59:44.696230 10096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:59:44.696237 10099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:59:44.696249 10097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:59:44.696564 10091 server_base.cc:1034] running on GCE node
I20260924 04:59:44.696798 10091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:59:44.697036 10091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:59:44.698192 10091 hybrid_clock.cc:648] HybridClock initialized: now 1790225984698173 us; error 29 us; skew 500 ppm
I20260924 04:59:44.699379 10091 webserver.cc:460] Webserver started at http://127.0.98.62:38867/ using document root <none> and password file <none>
I20260924 04:59:44.699597 10091 fs_manager.cc:359] Metadata directory not provided
I20260924 04:59:44.699646 10091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:59:44.699782 10091 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:59:44.700795 10091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225929848825-392-0/minicluster-data/master-0/data/instance:
uuid: "b7c63e3745784750ae382cd8d78b25e3"
format_stamp: "Formatted at 2026-09-24 04:59:44 on dist-test-slave-70st"
server_key: "cb770513dee9d848b486a4b148b9ed87"
server_key_iv: "804141deb75e8b7067c9eaa748306a78"
server_key_version: "encryptionkey@0"
I20260924 04:59:44.701112 10091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225929848825-392-0/minicluster-data/master-0/wal/instance:
uuid: "b7c63e3745784750ae382cd8d78b25e3"
format_stamp: "Formatted at 2026-09-24 04:59:44 on dist-test-slave-70st"
server_key: "cb770513dee9d848b486a4b148b9ed87"
server_key_iv: "804141deb75e8b7067c9eaa748306a78"
server_key_version: "encryptionkey@0"
I20260924 04:59:44.702368 10091 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 04:59:44.702972 10105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:44.703122 10091 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 04:59:44.703199 10091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225929848825-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225929848825-392-0/minicluster-data/master-0/wal
uuid: "b7c63e3745784750ae382cd8d78b25e3"
format_stamp: "Formatted at 2026-09-24 04:59:44 on dist-test-slave-70st"
server_key: "cb770513dee9d848b486a4b148b9ed87"
server_key_iv: "804141deb75e8b7067c9eaa748306a78"
server_key_version: "encryptionkey@0"
I20260924 04:59:44.703259 10091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225929848825-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225929848825-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225929848825-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:59:44.706506 10091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:59:44.706753 10091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:59:44.706876 10091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:59:44.710378 10091 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40609
I20260924 04:59:44.710412 10157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40609 every 8 connection(s)
I20260924 04:59:44.710716 10091 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:59:44.711338 10158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:59:44.714072 10158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7c63e3745784750ae382cd8d78b25e3: Bootstrap starting.
I20260924 04:59:44.714597 10158 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b7c63e3745784750ae382cd8d78b25e3: Neither blocks nor log segments found. Creating new log.
I20260924 04:59:44.714824 10158 log.cc:824] T 00000000000000000000000000000000 P b7c63e3745784750ae382cd8d78b25e3: Log is configured to *not* fsync() on all Append() calls
I20260924 04:59:44.715411 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 10091
I20260924 04:59:44.715476 10158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7c63e3745784750ae382cd8d78b25e3: No bootstrap required, opened a new log
I20260924 04:59:44.715487 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225929848825-392-0/minicluster-data/master-0/wal/instance
I20260924 04:59:44.715634 392 external_mini_cluster.cc:1428] Setting key e15d2f39f4c3f2629eac8e9b6293c7ad
I20260924 04:59:44.717388 10158 raft_consensus.cc:348] T 00000000000000000000000000000000 P b7c63e3745784750ae382cd8d78b25e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7c63e3745784750ae382cd8d78b25e3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40609 } }
I20260924 04:59:44.717589 10158 raft_consensus.cc:374] T 00000000000000000000000000000000 P b7c63e3745784750ae382cd8d78b25e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:59:44.717662 10158 raft_consensus.cc:729] T 00000000000000000000000000000000 P b7c63e3745784750ae382cd8d78b25e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7c63e3745784750ae382cd8d78b25e3, State: Initialized, Role: FOLLOWER
I20260924 04:59:44.717829 10158 consensus_queue.cc:260] T 00000000000000000000000000000000 P b7c63e3745784750ae382cd8d78b25e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7c63e3745784750ae382cd8d78b25e3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40609 } }
I20260924 04:59:44.717924 10158 raft_consensus.cc:388] T 00000000000000000000000000000000 P b7c63e3745784750ae382cd8d78b25e3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:59:44.717988 10158 raft_consensus.cc:482] T 00000000000000000000000000000000 P b7c63e3745784750ae382cd8d78b25e3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:59:44.718039 10158 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b7c63e3745784750ae382cd8d78b25e3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:59:44.718713 10158 raft_consensus.cc:504] T 00000000000000000000000000000000 P b7c63e3745784750ae382cd8d78b25e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7c63e3745784750ae382cd8d78b25e3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40609 } }
I20260924 04:59:44.718758 10162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:44.716150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58826 (local address 127.0.98.62:40609)
0924 04:59:44.716370 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:44.716379 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:44.716414 (+ 35us) server_negotiation.cc:407] Connection header received
0924 04:59:44.717345 (+ 931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:44.717354 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:44.717407 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:44.717583 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:44.718262 (+ 679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:44.718266 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:44.718266 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:59:44.718271 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:44.718315 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:59:44.718329 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:44.718401 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:44.718512 (+ 111us) server_negotiation.cc:299] Negotiation successful
0924 04:59:44.718570 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":84,"threads_started":1}
I20260924 04:59:44.718819 10158 leader_election.cc:302] T 00000000000000000000000000000000 P b7c63e3745784750ae382cd8d78b25e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b7c63e3745784750ae382cd8d78b25e3; no voters:
I20260924 04:59:44.718947 10158 leader_election.cc:288] T 00000000000000000000000000000000 P b7c63e3745784750ae382cd8d78b25e3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:59:44.719211 10163 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b7c63e3745784750ae382cd8d78b25e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:59:44.719323 10163 raft_consensus.cc:686] T 00000000000000000000000000000000 P b7c63e3745784750ae382cd8d78b25e3 [term 1 LEADER]: Becoming Leader. State: Replica: b7c63e3745784750ae382cd8d78b25e3, State: Running, Role: LEADER
I20260924 04:59:44.719408 10163 consensus_queue.cc:237] T 00000000000000000000000000000000 P b7c63e3745784750ae382cd8d78b25e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7c63e3745784750ae382cd8d78b25e3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40609 } }
I20260924 04:59:44.719609 10158 sys_catalog.cc:564] T 00000000000000000000000000000000 P b7c63e3745784750ae382cd8d78b25e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:59:44.719815 10163 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7c63e3745784750ae382cd8d78b25e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b7c63e3745784750ae382cd8d78b25e3. Latest consensus state: current_term: 1 leader_uuid: "b7c63e3745784750ae382cd8d78b25e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7c63e3745784750ae382cd8d78b25e3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40609 } } }
I20260924 04:59:44.719846 10164 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7c63e3745784750ae382cd8d78b25e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b7c63e3745784750ae382cd8d78b25e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7c63e3745784750ae382cd8d78b25e3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40609 } } }
I20260924 04:59:44.719985 10164 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7c63e3745784750ae382cd8d78b25e3 [sys.catalog]: This master's current role is: LEADER
I20260924 04:59:44.719970 10163 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7c63e3745784750ae382cd8d78b25e3 [sys.catalog]: This master's current role is: LEADER
I20260924 04:59:44.720436 10169 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:59:44.720911 10169 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:59:44.722379 10169 catalog_manager.cc:1347] Generated new cluster ID: 3452c319e9ea44b482ceddaa0e543add
I20260924 04:59:44.722429 10169 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 04:59:44.735793 10169 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:59:44.735994 10169 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:59:44.742959 10169 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b7c63e3745784750ae382cd8d78b25e3: Generated new TSK 0
I20260924 04:59:44.743148 10169 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 04:59:44.748708 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225929848825-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225929848825-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225929848825-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225929848825-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40609
--builtin_ntp_servers=127.0.98.20:46135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 04:59:44.849728 10182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:59:44.849924 10182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:59:44.849946 10182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:59:44.851097 10182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:59:44.851136 10182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:59:44.851161 10182 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:59:44.852490 10182 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225929848825-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225929848825-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225929848825-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40609
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10182
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225929848825-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 04:59:44.852710 10182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:59:44.852988 10182 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:59:44.857637 10188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:59:44.857661 10187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:59:44.857791 10190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:59:44.857730 10182 server_base.cc:1034] running on GCE node
I20260924 04:59:44.858008 10182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:59:44.858224 10182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:59:44.859361 10182 hybrid_clock.cc:648] HybridClock initialized: now 1790225984859341 us; error 41 us; skew 500 ppm
I20260924 04:59:44.860546 10182 webserver.cc:460] Webserver started at http://127.0.98.1:43929/ using document root <none> and password file <none>
I20260924 04:59:44.860772 10182 fs_manager.cc:359] Metadata directory not provided
I20260924 04:59:44.860819 10182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:59:44.860940 10182 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:59:44.862007 10182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225929848825-392-0/minicluster-data/ts-0/data/instance:
uuid: "405df2be639144d782dfb4c8fd9bea4d"
format_stamp: "Formatted at 2026-09-24 04:59:44 on dist-test-slave-70st"
server_key: "61d64d3a7e560f76e1181f14d7512704"
server_key_iv: "75c7c2bc87999163f4bf08fe4cdff8f5"
server_key_version: "encryptionkey@0"
I20260924 04:59:44.862318 10182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225929848825-392-0/minicluster-data/ts-0/wal/instance:
uuid: "405df2be639144d782dfb4c8fd9bea4d"
format_stamp: "Formatted at 2026-09-24 04:59:44 on dist-test-slave-70st"
server_key: "61d64d3a7e560f76e1181f14d7512704"
server_key_iv: "75c7c2bc87999163f4bf08fe4cdff8f5"
server_key_version: "encryptionkey@0"
I20260924 04:59:44.863533 10182 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 04:59:44.864192 10196 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:44.864336 10182 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 04:59:44.864413 10182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225929848825-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225929848825-392-0/minicluster-data/ts-0/wal
uuid: "405df2be639144d782dfb4c8fd9bea4d"
format_stamp: "Formatted at 2026-09-24 04:59:44 on dist-test-slave-70st"
server_key: "61d64d3a7e560f76e1181f14d7512704"
server_key_iv: "75c7c2bc87999163f4bf08fe4cdff8f5"
server_key_version: "encryptionkey@0"
I20260924 04:59:44.864481 10182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225929848825-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225929848825-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225929848825-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:59:44.867617 10182 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:59:44.867866 10182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:59:44.867987 10182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:59:44.868165 10182 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:59:44.868432 10182 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:59:44.868462 10182 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:44.868490 10182 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:59:44.868511 10182 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:44.873710 10182 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38675
I20260924 04:59:44.873785 10309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38675 every 8 connection(s)
I20260924 04:59:44.874051 10182 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225929848825-392-0/minicluster-data/ts-0/data/info.pb
I20260924 04:59:44.876911 10162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:44.874750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40029 (local address 127.0.98.62:40609)
0924 04:59:44.874800 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:44.874802 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:44.875059 (+ 257us) server_negotiation.cc:407] Connection header received
0924 04:59:44.875540 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:44.875541 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:44.875548 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:44.875600 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:44.876681 (+ 1081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:44.876683 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:44.876683 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:59:44.876687 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:44.876703 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:59:44.876707 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:44.876756 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:44.876828 (+ 72us) server_negotiation.cc:299] Negotiation successful
0924 04:59:44.876841 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 04:59:44.877091 10312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:44.874743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40609 (local address 127.0.98.1:40029)
0924 04:59:44.874933 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:44.874974 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:44.875445 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:44.875665 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:44.875668 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:44.875675 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:44.876537 (+ 862us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:59:44.876538 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:44.876567 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:59:44.876594 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:59:44.876758 (+ 164us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:59:44.876763 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:59:44.876766 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 04:59:44.876833 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 04:59:44.876893 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":82,"threads_started":1}
I20260924 04:59:44.877637 10310 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40609
I20260924 04:59:44.877720 10310 heartbeater.cc:464] Registering TS with master...
I20260924 04:59:44.877830 10310 heartbeater.cc:511] Master 127.0.98.62:40609 requested a full tablet report, sending...
I20260924 04:59:44.878276 10122 ts_manager.cc:194] Registered new tserver with Master: 405df2be639144d782dfb4c8fd9bea4d (127.0.98.1:38675)
I20260924 04:59:44.883746 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 10182
I20260924 04:59:44.883823 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225929848825-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:59:44.883996 392 external_mini_cluster.cc:1428] Setting key 4bfc6710547c255ccb32353efd7b0d2e
I20260924 04:59:44.884828 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225929848825-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225929848825-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225929848825-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225929848825-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40609
--builtin_ntp_servers=127.0.98.20:46135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 04:59:44.963793 10313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:59:44.963990 10313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:59:44.964027 10313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:59:44.965130 10313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:59:44.965170 10313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:59:44.965204 10313 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:59:44.966782 10313 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225929848825-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225929848825-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225929848825-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40609
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10313
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225929848825-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 04:59:44.967008 10313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:59:44.967269 10313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:59:44.972048 10319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:59:44.972178 10318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:59:44.972218 10313 server_base.cc:1034] running on GCE node
W20260924 04:59:44.972077 10321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:59:44.972494 10313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:59:44.972707 10313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:59:44.973838 10313 hybrid_clock.cc:648] HybridClock initialized: now 1790225984973830 us; error 35 us; skew 500 ppm
I20260924 04:59:44.975057 10313 webserver.cc:460] Webserver started at http://127.0.98.2:39703/ using document root <none> and password file <none>
I20260924 04:59:44.975297 10313 fs_manager.cc:359] Metadata directory not provided
I20260924 04:59:44.975342 10313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:59:44.975464 10313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:59:44.976522 10313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225929848825-392-0/minicluster-data/ts-1/data/instance:
uuid: "a508b4cbcb1c46e8b7b36837bfde11c5"
format_stamp: "Formatted at 2026-09-24 04:59:44 on dist-test-slave-70st"
server_key: "715bb1a5cba3119ccf1c5108face83ba"
server_key_iv: "c9f5e4ae49acde0c62ae2968c9141910"
server_key_version: "encryptionkey@0"
I20260924 04:59:44.976847 10313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225929848825-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a508b4cbcb1c46e8b7b36837bfde11c5"
format_stamp: "Formatted at 2026-09-24 04:59:44 on dist-test-slave-70st"
server_key: "715bb1a5cba3119ccf1c5108face83ba"
server_key_iv: "c9f5e4ae49acde0c62ae2968c9141910"
server_key_version: "encryptionkey@0"
I20260924 04:59:44.978173 10313 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 04:59:44.978838 10327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:44.978986 10313 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 04:59:44.979060 10313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225929848825-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225929848825-392-0/minicluster-data/ts-1/wal
uuid: "a508b4cbcb1c46e8b7b36837bfde11c5"
format_stamp: "Formatted at 2026-09-24 04:59:44 on dist-test-slave-70st"
server_key: "715bb1a5cba3119ccf1c5108face83ba"
server_key_iv: "c9f5e4ae49acde0c62ae2968c9141910"
server_key_version: "encryptionkey@0"
I20260924 04:59:44.979125 10313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225929848825-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225929848825-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225929848825-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:59:44.982363 10313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:59:44.982597 10313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:59:44.982707 10313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:59:44.982892 10313 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:59:44.983158 10313 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:59:44.983188 10313 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:44.983215 10313 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:59:44.983238 10313 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:44.988471 10313 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37475
I20260924 04:59:44.988523 10440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37475 every 8 connection(s)
I20260924 04:59:44.988811 10313 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225929848825-392-0/minicluster-data/ts-1/data/info.pb
I20260924 04:59:44.991689 10162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:44.989476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38343 (local address 127.0.98.62:40609)
0924 04:59:44.989519 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:44.989521 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:44.989745 (+ 224us) server_negotiation.cc:407] Connection header received
0924 04:59:44.990233 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:44.990234 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:44.990242 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:44.990300 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:44.991451 (+ 1151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:44.991453 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:44.991453 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:59:44.991458 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:44.991477 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:59:44.991482 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:44.991531 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:44.991606 (+ 75us) server_negotiation.cc:299] Negotiation successful
0924 04:59:44.991621 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 04:59:44.991868 10443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:44.989471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40609 (local address 127.0.98.2:38343)
0924 04:59:44.989626 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:44.989658 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:44.990128 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:44.990372 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:44.990376 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:44.990385 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:44.991288 (+ 903us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:59:44.991289 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:44.991319 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:59:44.991353 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:59:44.991533 (+ 180us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:59:44.991537 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:59:44.991540 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 04:59:44.991608 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 04:59:44.991668 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":70,"threads_started":1}
I20260924 04:59:44.992292 10441 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40609
I20260924 04:59:44.992369 10441 heartbeater.cc:464] Registering TS with master...
I20260924 04:59:44.992504 10441 heartbeater.cc:511] Master 127.0.98.62:40609 requested a full tablet report, sending...
I20260924 04:59:44.992835 10122 ts_manager.cc:194] Registered new tserver with Master: a508b4cbcb1c46e8b7b36837bfde11c5 (127.0.98.2:37475)
I20260924 04:59:44.998916 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 10313
I20260924 04:59:44.999002 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225929848825-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:59:44.999173 392 external_mini_cluster.cc:1428] Setting key 5b719b8fe1893bb6e5367b22d0e4a990
I20260924 04:59:45.000074 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225929848825-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225929848825-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225929848825-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225929848825-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40609
--builtin_ntp_servers=127.0.98.20:46135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 04:59:45.079074 10444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:59:45.079286 10444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:59:45.079308 10444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:59:45.080435 10444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:59:45.080474 10444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:59:45.080499 10444 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:59:45.081887 10444 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225929848825-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225929848825-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225929848825-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40609
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10444
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225929848825-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 04:59:45.082108 10444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:59:45.082405 10444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:59:45.087054 10450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:59:45.087044 10452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:59:45.087044 10449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:59:45.087447 10444 server_base.cc:1034] running on GCE node
I20260924 04:59:45.087653 10444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:59:45.087890 10444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:59:45.089023 10444 hybrid_clock.cc:648] HybridClock initialized: now 1790225985089013 us; error 29 us; skew 500 ppm
I20260924 04:59:45.090232 10444 webserver.cc:460] Webserver started at http://127.0.98.3:36999/ using document root <none> and password file <none>
I20260924 04:59:45.090451 10444 fs_manager.cc:359] Metadata directory not provided
I20260924 04:59:45.090497 10444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:59:45.090624 10444 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:59:45.091645 10444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225929848825-392-0/minicluster-data/ts-2/data/instance:
uuid: "8eb1af7a567b4046a9621c83cb9025a7"
format_stamp: "Formatted at 2026-09-24 04:59:45 on dist-test-slave-70st"
server_key: "549329efa251eb392163b61c0570cc4e"
server_key_iv: "6fa1bed7abecf005a455d48a07585517"
server_key_version: "encryptionkey@0"
I20260924 04:59:45.091984 10444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225929848825-392-0/minicluster-data/ts-2/wal/instance:
uuid: "8eb1af7a567b4046a9621c83cb9025a7"
format_stamp: "Formatted at 2026-09-24 04:59:45 on dist-test-slave-70st"
server_key: "549329efa251eb392163b61c0570cc4e"
server_key_iv: "6fa1bed7abecf005a455d48a07585517"
server_key_version: "encryptionkey@0"
I20260924 04:59:45.093205 10444 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 04:59:45.093931 10458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:45.094069 10444 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:45.094146 10444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225929848825-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225929848825-392-0/minicluster-data/ts-2/wal
uuid: "8eb1af7a567b4046a9621c83cb9025a7"
format_stamp: "Formatted at 2026-09-24 04:59:45 on dist-test-slave-70st"
server_key: "549329efa251eb392163b61c0570cc4e"
server_key_iv: "6fa1bed7abecf005a455d48a07585517"
server_key_version: "encryptionkey@0"
I20260924 04:59:45.094209 10444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225929848825-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225929848825-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225929848825-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:59:45.097481 10444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:59:45.097729 10444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:59:45.097851 10444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:59:45.098023 10444 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:59:45.098290 10444 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:59:45.098323 10444 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:45.098348 10444 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:59:45.098371 10444 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:45.103811 10444 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39459
I20260924 04:59:45.103873 10571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39459 every 8 connection(s)
I20260924 04:59:45.104183 10444 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225929848825-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:59:45.107317 10162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:45.104876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33671 (local address 127.0.98.62:40609)
0924 04:59:45.104962 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:45.104964 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:45.105182 (+ 218us) server_negotiation.cc:407] Connection header received
0924 04:59:45.105771 (+ 589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:45.105772 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:45.105781 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:45.105840 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:45.107031 (+ 1191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:45.107032 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:45.107032 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 04:59:45.107037 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:45.107054 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 04:59:45.107060 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:45.107114 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:45.107224 (+ 110us) server_negotiation.cc:299] Negotiation successful
0924 04:59:45.107243 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 04:59:45.107484 10574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:45.104852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40609 (local address 127.0.98.3:33671)
0924 04:59:45.105035 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:45.105068 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:45.105644 (+ 576us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:45.105895 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:45.105899 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:45.105907 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:45.106855 (+ 948us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 04:59:45.106855 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:45.106885 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 04:59:45.106914 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:59:45.107117 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:59:45.107122 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:59:45.107125 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 04:59:45.107202 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 04:59:45.107268 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":86,"threads_started":1}
I20260924 04:59:45.107932 10572 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40609
I20260924 04:59:45.108011 10572 heartbeater.cc:464] Registering TS with master...
I20260924 04:59:45.108134 10572 heartbeater.cc:511] Master 127.0.98.62:40609 requested a full tablet report, sending...
I20260924 04:59:45.108525 10122 ts_manager.cc:194] Registered new tserver with Master: 8eb1af7a567b4046a9621c83cb9025a7 (127.0.98.3:39459)
I20260924 04:59:45.114241 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 10444
I20260924 04:59:45.114328 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790225929848825-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:59:45.114487 392 external_mini_cluster.cc:1428] Setting key 7eb903c5887bc1130b499c362f5ae664
I20260924 04:59:45.115409 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 04:59:45.117116 10162 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 04:59:45.116668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58830 (local address 127.0.98.62:40609)
0924 04:59:45.116734 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:45.116735 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:45.116825 (+ 90us) server_negotiation.cc:407] Connection header received
0924 04:59:45.116881 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:45.116882 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:45.116888 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:45.116934 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:45.117050 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58830: BlockingRecv error: recv got EOF from 127.0.0.1:58830 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 04:59:45.117177 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 10182
I20260924 04:59:45.121975 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 10313
I20260924 04:59:45.127408 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 10444
I20260924 04:59:45.131356 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 10091
2026-09-24T04:59:45Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (543 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225929848825-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:59:45 dist-test-slave-70st krb5kdc[10582](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:59:45 dist-test-slave-70st krb5kdc[10582](info): set up 2 sockets
Sep 24 04:59:45 dist-test-slave-70st krb5kdc[10582](info): commencing operation
krb5kdc: starting...
W20260924 04:59:46.077961 392 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225929848825-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225929848825-392-0/krb5kdc/test-user.keytab.
Sep 24 04:59:46 dist-test-slave-70st krb5kdc[10582](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225986, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:59:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:59:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 04:59:46.188046 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225929848825-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225929848825-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225929848825-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40709
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40709
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:59:46.266146 10595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:59:46.266364 10595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:59:46.266394 10595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:59:46.267541 10595 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 04:59:46.267588 10595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:59:46.267606 10595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:59:46.267621 10595 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 04:59:46.267637 10595 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 04:59:46.269125 10595 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225929848825-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225929848825-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40709
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10595
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225929848825-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 04:59:46.269368 10595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:59:46.269593 10595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:59:46.273944 10600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:59:46.274049 10595 server_base.cc:1034] running on GCE node
W20260924 04:59:46.273955 10601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:59:46.273958 10603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:59:46.274392 10595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:59:46.274674 10595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:59:46.275820 10595 hybrid_clock.cc:648] HybridClock initialized: now 1790225986275807 us; error 27 us; skew 500 ppm
Sep 24 04:59:46 dist-test-slave-70st krb5kdc[10582](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225986, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:59:46.286919 10595 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 04:59:46.287322 10595 webserver.cc:460] Webserver started at http://127.0.98.62:35505/ using document root <none> and password file <none>
I20260924 04:59:46.287526 10595 fs_manager.cc:359] Metadata directory not provided
I20260924 04:59:46.287570 10595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:59:46.287688 10595 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:59:46.288692 10595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225929848825-392-0/minicluster-data/master-0/data/instance:
uuid: "16a60adc553b438b8b26acfa4f84ce1c"
format_stamp: "Formatted at 2026-09-24 04:59:46 on dist-test-slave-70st"
server_key: "0540cf9ef63b41b5ba420087db90c9fb"
server_key_iv: "23edcc08ee999e90627e518bfc91dccb"
server_key_version: "encryptionkey@0"
I20260924 04:59:46.289004 10595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225929848825-392-0/minicluster-data/master-0/wal/instance:
uuid: "16a60adc553b438b8b26acfa4f84ce1c"
format_stamp: "Formatted at 2026-09-24 04:59:46 on dist-test-slave-70st"
server_key: "0540cf9ef63b41b5ba420087db90c9fb"
server_key_iv: "23edcc08ee999e90627e518bfc91dccb"
server_key_version: "encryptionkey@0"
I20260924 04:59:46.290279 10595 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 04:59:46.290987 10610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:46.291147 10595 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 04:59:46.291217 10595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225929848825-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225929848825-392-0/minicluster-data/master-0/wal
uuid: "16a60adc553b438b8b26acfa4f84ce1c"
format_stamp: "Formatted at 2026-09-24 04:59:46 on dist-test-slave-70st"
server_key: "0540cf9ef63b41b5ba420087db90c9fb"
server_key_iv: "23edcc08ee999e90627e518bfc91dccb"
server_key_version: "encryptionkey@0"
I20260924 04:59:46.291275 10595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225929848825-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225929848825-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225929848825-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:59:46.307147 10595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:59:46.309002 10595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:59:46.309116 10595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:59:46.312852 10595 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40709
I20260924 04:59:46.312909 10662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40709 every 8 connection(s)
I20260924 04:59:46.313203 10595 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
I20260924 04:59:46.314132 10663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:59:46.316336 10663 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16a60adc553b438b8b26acfa4f84ce1c: Bootstrap starting.
I20260924 04:59:46.316865 10663 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 16a60adc553b438b8b26acfa4f84ce1c: Neither blocks nor log segments found. Creating new log.
I20260924 04:59:46.317107 10663 log.cc:824] T 00000000000000000000000000000000 P 16a60adc553b438b8b26acfa4f84ce1c: Log is configured to *not* fsync() on all Append() calls
I20260924 04:59:46.317740 10663 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16a60adc553b438b8b26acfa4f84ce1c: No bootstrap required, opened a new log
I20260924 04:59:46.319232 10663 raft_consensus.cc:348] T 00000000000000000000000000000000 P 16a60adc553b438b8b26acfa4f84ce1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16a60adc553b438b8b26acfa4f84ce1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40709 } }
I20260924 04:59:46.319365 10663 raft_consensus.cc:374] T 00000000000000000000000000000000 P 16a60adc553b438b8b26acfa4f84ce1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:59:46.319399 10663 raft_consensus.cc:729] T 00000000000000000000000000000000 P 16a60adc553b438b8b26acfa4f84ce1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16a60adc553b438b8b26acfa4f84ce1c, State: Initialized, Role: FOLLOWER
I20260924 04:59:46.319545 10663 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16a60adc553b438b8b26acfa4f84ce1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16a60adc553b438b8b26acfa4f84ce1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40709 } }
I20260924 04:59:46.319621 10663 raft_consensus.cc:388] T 00000000000000000000000000000000 P 16a60adc553b438b8b26acfa4f84ce1c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 04:59:46.319651 10663 raft_consensus.cc:482] T 00000000000000000000000000000000 P 16a60adc553b438b8b26acfa4f84ce1c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 04:59:46.319696 10663 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 16a60adc553b438b8b26acfa4f84ce1c [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:59:46.320422 10663 raft_consensus.cc:504] T 00000000000000000000000000000000 P 16a60adc553b438b8b26acfa4f84ce1c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16a60adc553b438b8b26acfa4f84ce1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40709 } }
I20260924 04:59:46.320520 10663 leader_election.cc:302] T 00000000000000000000000000000000 P 16a60adc553b438b8b26acfa4f84ce1c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 16a60adc553b438b8b26acfa4f84ce1c; no voters:
I20260924 04:59:46.320674 10663 leader_election.cc:288] T 00000000000000000000000000000000 P 16a60adc553b438b8b26acfa4f84ce1c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 04:59:46.320700 10666 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 16a60adc553b438b8b26acfa4f84ce1c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:59:46.320815 10666 raft_consensus.cc:686] T 00000000000000000000000000000000 P 16a60adc553b438b8b26acfa4f84ce1c [term 1 LEADER]: Becoming Leader. State: Replica: 16a60adc553b438b8b26acfa4f84ce1c, State: Running, Role: LEADER
I20260924 04:59:46.320902 10663 sys_catalog.cc:564] T 00000000000000000000000000000000 P 16a60adc553b438b8b26acfa4f84ce1c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 04:59:46.320945 10666 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16a60adc553b438b8b26acfa4f84ce1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16a60adc553b438b8b26acfa4f84ce1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40709 } }
I20260924 04:59:46.321276 10668 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16a60adc553b438b8b26acfa4f84ce1c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16a60adc553b438b8b26acfa4f84ce1c. Latest consensus state: current_term: 1 leader_uuid: "16a60adc553b438b8b26acfa4f84ce1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16a60adc553b438b8b26acfa4f84ce1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40709 } } }
I20260924 04:59:46.321266 10667 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16a60adc553b438b8b26acfa4f84ce1c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "16a60adc553b438b8b26acfa4f84ce1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16a60adc553b438b8b26acfa4f84ce1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40709 } } }
I20260924 04:59:46.321457 10667 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16a60adc553b438b8b26acfa4f84ce1c [sys.catalog]: This master's current role is: LEADER
I20260924 04:59:46.321669 10668 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16a60adc553b438b8b26acfa4f84ce1c [sys.catalog]: This master's current role is: LEADER
I20260924 04:59:46.321784 10674 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 04:59:46.322252 10674 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 04:59:46.322356 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 10595
I20260924 04:59:46.322456 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225929848825-392-0/minicluster-data/master-0/wal/instance
I20260924 04:59:46.322597 392 external_mini_cluster.cc:1428] Setting key 2f6ae5b4dc116b9f90682aadf1bae3d1
I20260924 04:59:46.324052 10674 catalog_manager.cc:1347] Generated new cluster ID: 547bdb157dad47108b9de86114e5a90a
I20260924 04:59:46.324105 10674 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 04:59:46 dist-test-slave-70st krb5kdc[10582](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225986, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:59:46.331333 10685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:46.322999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45956 (local address 127.0.98.62:40709)
0924 04:59:46.323150 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:46.323159 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:46.323331 (+ 172us) server_negotiation.cc:407] Connection header received
0924 04:59:46.323460 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:46.323467 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:46.323521 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:46.323651 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:46.324240 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:46.325117 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:46.325876 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:46.326280 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:46.326373 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:46.328445 (+ 2072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:46.328449 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:46.328449 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:59:46.328461 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:46.329981 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:46.330280 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:46.330284 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:46.330284 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:46.330373 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:46.330550 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:46.330551 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:46.330551 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:46.330772 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:46.330909 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:46.331019 (+ 110us) server_negotiation.cc:299] Negotiation successful
0924 04:59:46.331144 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":84,"threads_started":1}
I20260924 04:59:46.340387 10674 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 04:59:46.340871 10674 catalog_manager.cc:1504] Loading token signing keys...
I20260924 04:59:46.350059 10674 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 16a60adc553b438b8b26acfa4f84ce1c: Generated new TSK 0
I20260924 04:59:46.350241 10674 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225929848825-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225929848825-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 04:59:46.379928 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225929848825-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225929848825-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225929848825-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225929848825-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40709
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225929848825-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:59:46.476123 10689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:59:46.476321 10689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:59:46.476343 10689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:59:46.477437 10689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:59:46.477486 10689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:59:46.477514 10689 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 04:59:46.478912 10689 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225929848825-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225929848825-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225929848825-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225929848825-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10689
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225929848825-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 04:59:46.479101 10689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:59:46.479270 10689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 04:59:46.483971 10689 server_base.cc:1034] running on GCE node
W20260924 04:59:46.484005 10695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:59:46.483954 10694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:59:46.483955 10697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:59:46.484402 10689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:59:46.484622 10689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:59:46.485776 10689 hybrid_clock.cc:648] HybridClock initialized: now 1790225986485760 us; error 34 us; skew 500 ppm
Sep 24 04:59:46 dist-test-slave-70st krb5kdc[10582](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225986, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:59:46.488097 10689 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 04:59:46.488471 10689 webserver.cc:460] Webserver started at http://127.0.98.1:35011/ using document root <none> and password file <none>
I20260924 04:59:46.488648 10689 fs_manager.cc:359] Metadata directory not provided
I20260924 04:59:46.488686 10689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:59:46.488785 10689 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:59:46.489797 10689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225929848825-392-0/minicluster-data/ts-0/data/instance:
uuid: "055bdbb5b4bb417f9f338d61d056def2"
format_stamp: "Formatted at 2026-09-24 04:59:46 on dist-test-slave-70st"
server_key: "7cc59dd0329ac6aa06002364501505ba"
server_key_iv: "1e3c18c6fd7680250b6aa468ccee3668"
server_key_version: "encryptionkey@0"
I20260924 04:59:46.490080 10689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225929848825-392-0/minicluster-data/ts-0/wal/instance:
uuid: "055bdbb5b4bb417f9f338d61d056def2"
format_stamp: "Formatted at 2026-09-24 04:59:46 on dist-test-slave-70st"
server_key: "7cc59dd0329ac6aa06002364501505ba"
server_key_iv: "1e3c18c6fd7680250b6aa468ccee3668"
server_key_version: "encryptionkey@0"
I20260924 04:59:46.491207 10689 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 04:59:46.491806 10704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:46.491948 10689 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 04:59:46.492027 10689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225929848825-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225929848825-392-0/minicluster-data/ts-0/wal
uuid: "055bdbb5b4bb417f9f338d61d056def2"
format_stamp: "Formatted at 2026-09-24 04:59:46 on dist-test-slave-70st"
server_key: "7cc59dd0329ac6aa06002364501505ba"
server_key_iv: "1e3c18c6fd7680250b6aa468ccee3668"
server_key_version: "encryptionkey@0"
I20260924 04:59:46.492089 10689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225929848825-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225929848825-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225929848825-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:59:46.516728 10689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:59:46.518944 10689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:59:46.519083 10689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:59:46.519299 10689 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:59:46.519593 10689 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:59:46.519627 10689 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:46.519656 10689 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:59:46.519680 10689 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:46.525238 10689 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40671
I20260924 04:59:46.525308 10817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40671 every 8 connection(s)
I20260924 04:59:46.525646 10689 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225929848825-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 04:59:46 dist-test-slave-70st krb5kdc[10582](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225986, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:59:46.533946 10685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:46.526333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38129 (local address 127.0.98.62:40709)
0924 04:59:46.526377 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:46.526379 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:46.526590 (+ 211us) server_negotiation.cc:407] Connection header received
0924 04:59:46.527102 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:46.527103 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:46.527118 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:46.527162 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:46.528091 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:46.528592 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:46.529580 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:46.529789 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:46.529844 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:46.531812 (+ 1968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:46.531813 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:46.531813 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:59:46.531820 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:46.532870 (+ 1050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:46.533196 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:46.533197 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:46.533197 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:46.533241 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:46.533453 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:46.533454 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:46.533454 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:46.533555 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:46.533603 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:46.533818 (+ 215us) server_negotiation.cc:299] Negotiation successful
0924 04:59:46.533882 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 04:59:46.534109 10820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:46.526316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40709 (local address 127.0.98.1:38129)
0924 04:59:46.526481 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:46.526516 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:46.527006 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:46.527249 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:46.527253 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:46.527808 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:46.528001 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:46.528006 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:46.528670 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:46.528670 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:46.529508 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:46.529511 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:46.529856 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:46.529856 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:46.529983 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:46.530433 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:46.530440 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:46.531716 (+ 1276us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:59:46.532935 (+ 1219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:46.532940 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:46.532942 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:46.533141 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:46.533290 (+ 149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:46.533291 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:46.533291 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:46.533381 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:46.533600 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:59:46.533603 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:59:46.533713 (+ 110us) client_negotiation.cc:769] Sending connection context
0924 04:59:46.533800 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 04:59:46.533913 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":85,"threads_started":1}
I20260924 04:59:46.534565 10818 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40709
I20260924 04:59:46.534647 10818 heartbeater.cc:464] Registering TS with master...
I20260924 04:59:46.534809 10818 heartbeater.cc:511] Master 127.0.98.62:40709 requested a full tablet report, sending...
I20260924 04:59:46.535163 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 10689
I20260924 04:59:46.535252 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225929848825-392-0/minicluster-data/ts-0/wal/instance
I20260924 04:59:46.535327 10627 ts_manager.cc:194] Registered new tserver with Master: 055bdbb5b4bb417f9f338d61d056def2 (127.0.98.1:40671)
I20260924 04:59:46.535440 392 external_mini_cluster.cc:1428] Setting key 56efb7fa18b0ec802c2a094e7a3f2f90
I20260924 04:59:46.536007 10627 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:38129
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225929848825-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225929848825-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 04:59:46.557538 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225929848825-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225929848825-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225929848825-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225929848825-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40709
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225929848825-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:59:46.634836 10823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:59:46.635077 10823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:59:46.635110 10823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:59:46.636272 10823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:59:46.636317 10823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:59:46.636351 10823 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 04:59:46.637853 10823 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225929848825-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225929848825-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225929848825-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225929848825-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10823
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225929848825-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 04:59:46.638098 10823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:59:46.638312 10823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:59:46.643136 10831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:59:46.643123 10828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:59:46.643153 10823 server_base.cc:1034] running on GCE node
W20260924 04:59:46.643123 10829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:59:46.643553 10823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:59:46.643802 10823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:59:46.644954 10823 hybrid_clock.cc:648] HybridClock initialized: now 1790225986644942 us; error 34 us; skew 500 ppm
Sep 24 04:59:46 dist-test-slave-70st krb5kdc[10582](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225986, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:59:46.647500 10823 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 04:59:46.647964 10823 webserver.cc:460] Webserver started at http://127.0.98.2:44133/ using document root <none> and password file <none>
I20260924 04:59:46.648180 10823 fs_manager.cc:359] Metadata directory not provided
I20260924 04:59:46.648236 10823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:59:46.648370 10823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:59:46.649521 10823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225929848825-392-0/minicluster-data/ts-1/data/instance:
uuid: "d2c39ba5a76740229afa6c481ac27d5a"
format_stamp: "Formatted at 2026-09-24 04:59:46 on dist-test-slave-70st"
server_key: "c44fc037e56e7cfd6abd643f2fc55924"
server_key_iv: "2b74b2d926bddf8dbc5197091c16f9e5"
server_key_version: "encryptionkey@0"
I20260924 04:59:46.649940 10823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225929848825-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d2c39ba5a76740229afa6c481ac27d5a"
format_stamp: "Formatted at 2026-09-24 04:59:46 on dist-test-slave-70st"
server_key: "c44fc037e56e7cfd6abd643f2fc55924"
server_key_iv: "2b74b2d926bddf8dbc5197091c16f9e5"
server_key_version: "encryptionkey@0"
I20260924 04:59:46.651285 10823 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 04:59:46.651965 10838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:46.652119 10823 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 04:59:46.652230 10823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225929848825-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225929848825-392-0/minicluster-data/ts-1/wal
uuid: "d2c39ba5a76740229afa6c481ac27d5a"
format_stamp: "Formatted at 2026-09-24 04:59:46 on dist-test-slave-70st"
server_key: "c44fc037e56e7cfd6abd643f2fc55924"
server_key_iv: "2b74b2d926bddf8dbc5197091c16f9e5"
server_key_version: "encryptionkey@0"
I20260924 04:59:46.652303 10823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225929848825-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225929848825-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225929848825-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:59:46.662912 10823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:59:46.665203 10823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:59:46.665344 10823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:59:46.665606 10823 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:59:46.665932 10823 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:59:46.665967 10823 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:46.666004 10823 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:59:46.666031 10823 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:46.671975 10823 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44831
I20260924 04:59:46.672010 10951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44831 every 8 connection(s)
I20260924 04:59:46.672346 10823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225929848825-392-0/minicluster-data/ts-1/data/info.pb
Sep 24 04:59:46 dist-test-slave-70st krb5kdc[10582](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225986, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:59:46.680784 10685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:46.673011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53723 (local address 127.0.98.62:40709)
0924 04:59:46.673055 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:46.673056 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:46.673249 (+ 193us) server_negotiation.cc:407] Connection header received
0924 04:59:46.673800 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:46.673801 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:46.673813 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:46.673860 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:46.674782 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:46.675300 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:46.676256 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:46.676459 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:46.676513 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:46.678562 (+ 2049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:46.678564 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:46.678564 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:59:46.678571 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:46.679612 (+ 1041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:46.679980 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:46.679981 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:46.679981 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:46.680040 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:46.680235 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:46.680236 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:46.680236 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:46.680344 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:46.680395 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:46.680646 (+ 251us) server_negotiation.cc:299] Negotiation successful
0924 04:59:46.680721 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 04:59:46.680999 10954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:46.672980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40709 (local address 127.0.98.2:53723)
0924 04:59:46.673126 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:46.673161 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:46.673703 (+ 542us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:46.673942 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:46.673946 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:46.674497 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:46.674689 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:46.674693 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:46.675390 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:46.675391 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:46.676174 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:46.676176 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:46.676524 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:46.676525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:46.676639 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:46.677092 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:46.677100 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:46.678478 (+ 1378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:59:46.679682 (+ 1204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:46.679688 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:46.679690 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:46.679897 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:46.680098 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:46.680098 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:46.680099 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:46.680186 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:46.680389 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:59:46.680392 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:59:46.680518 (+ 126us) client_negotiation.cc:769] Sending connection context
0924 04:59:46.680595 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 04:59:46.680708 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":90,"threads_started":1}
I20260924 04:59:46.681460 10952 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40709
I20260924 04:59:46.681547 10952 heartbeater.cc:464] Registering TS with master...
I20260924 04:59:46.681695 10952 heartbeater.cc:511] Master 127.0.98.62:40709 requested a full tablet report, sending...
I20260924 04:59:46.682024 10627 ts_manager.cc:194] Registered new tserver with Master: d2c39ba5a76740229afa6c481ac27d5a (127.0.98.2:44831)
I20260924 04:59:46.682333 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 10823
I20260924 04:59:46.682379 10627 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53723
I20260924 04:59:46.682423 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225929848825-392-0/minicluster-data/ts-1/wal/instance
I20260924 04:59:46.682610 392 external_mini_cluster.cc:1428] Setting key ee65ea1dcf4456d740974e1505ef730e
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225929848825-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225929848825-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 04:59:46.705981 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225929848825-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225929848825-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225929848825-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225929848825-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40709
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225929848825-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 04:59:46.784240 10957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 04:59:46.784451 10957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 04:59:46.784471 10957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 04:59:46.785640 10957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 04:59:46.785681 10957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 04:59:46.785704 10957 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 04:59:46.787106 10957 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225929848825-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225929848825-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225929848825-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225929848825-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10957
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225929848825-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 04:59:46.787312 10957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 04:59:46.787533 10957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 04:59:46.792212 10962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:59:46.792241 10965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 04:59:46.792240 10963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 04:59:46.792456 10957 server_base.cc:1034] running on GCE node
I20260924 04:59:46.792659 10957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 04:59:46.792883 10957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 04:59:46.794025 10957 hybrid_clock.cc:648] HybridClock initialized: now 1790225986794009 us; error 30 us; skew 500 ppm
Sep 24 04:59:46 dist-test-slave-70st krb5kdc[10582](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225986, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 04:59:46.796553 10957 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 04:59:46.796945 10957 webserver.cc:460] Webserver started at http://127.0.98.3:44751/ using document root <none> and password file <none>
I20260924 04:59:46.797138 10957 fs_manager.cc:359] Metadata directory not provided
I20260924 04:59:46.797184 10957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 04:59:46.797323 10957 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 04:59:46.798429 10957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225929848825-392-0/minicluster-data/ts-2/data/instance:
uuid: "8ee31b7a91ac4e88aaf53c23c9ae90b3"
format_stamp: "Formatted at 2026-09-24 04:59:46 on dist-test-slave-70st"
server_key: "1a3ad8d753b69174f32a81d1cc4a7398"
server_key_iv: "083bd6809028a3a24975d0eb62dc88fe"
server_key_version: "encryptionkey@0"
I20260924 04:59:46.798805 10957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225929848825-392-0/minicluster-data/ts-2/wal/instance:
uuid: "8ee31b7a91ac4e88aaf53c23c9ae90b3"
format_stamp: "Formatted at 2026-09-24 04:59:46 on dist-test-slave-70st"
server_key: "1a3ad8d753b69174f32a81d1cc4a7398"
server_key_iv: "083bd6809028a3a24975d0eb62dc88fe"
server_key_version: "encryptionkey@0"
I20260924 04:59:46.800016 10957 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 04:59:46.800716 10972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:46.800846 10957 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 04:59:46.800921 10957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225929848825-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225929848825-392-0/minicluster-data/ts-2/wal
uuid: "8ee31b7a91ac4e88aaf53c23c9ae90b3"
format_stamp: "Formatted at 2026-09-24 04:59:46 on dist-test-slave-70st"
server_key: "1a3ad8d753b69174f32a81d1cc4a7398"
server_key_iv: "083bd6809028a3a24975d0eb62dc88fe"
server_key_version: "encryptionkey@0"
I20260924 04:59:46.800985 10957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225929848825-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225929848825-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225929848825-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 04:59:46.808995 10957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 04:59:46.810966 10957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 04:59:46.811085 10957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 04:59:46.811275 10957 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 04:59:46.811558 10957 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 04:59:46.811590 10957 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:46.811612 10957 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 04:59:46.811636 10957 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 04:59:46.816788 10957 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42713
I20260924 04:59:46.816859 11085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42713 every 8 connection(s)
I20260924 04:59:46.817152 10957 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225929848825-392-0/minicluster-data/ts-2/data/info.pb
I20260924 04:59:46.820467 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 10957
I20260924 04:59:46.820581 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225929848825-392-0/minicluster-data/ts-2/wal/instance
I20260924 04:59:46.820745 392 external_mini_cluster.cc:1428] Setting key 3010f2fd799cbb5ed900abfbe66059b2
Sep 24 04:59:46 dist-test-slave-70st krb5kdc[10582](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225986, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 04:59:46.825845 10685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:46.817821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43135 (local address 127.0.98.62:40709)
0924 04:59:46.817867 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:46.817870 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:46.818067 (+ 197us) server_negotiation.cc:407] Connection header received
0924 04:59:46.818536 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:46.818537 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:46.818547 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:46.818587 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:46.819463 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:46.819947 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:46.821180 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:46.821408 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:46.821490 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:46.823563 (+ 2073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:46.823564 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:46.823565 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:59:46.823572 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:46.824636 (+ 1064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:46.824987 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:46.824988 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:46.824989 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:46.825039 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:46.825242 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:46.825243 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:46.825243 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:46.825358 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:46.825411 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:46.825679 (+ 268us) server_negotiation.cc:299] Negotiation successful
0924 04:59:46.825756 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 04:59:46.826012 11088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:46.817799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40709 (local address 127.0.98.3:43135)
0924 04:59:46.817952 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:46.817993 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:46.818444 (+ 451us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:46.818653 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:46.818657 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:46.819200 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0924 04:59:46.819399 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:46.819404 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:46.820007 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:46.820008 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:46.821026 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:46.821029 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:46.821495 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:46.821496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:46.821663 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:46.822179 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 04:59:46.822187 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 04:59:46.823480 (+ 1293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 04:59:46.824704 (+ 1224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:46.824710 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:46.824712 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:46.824925 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:46.825092 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 04:59:46.825092 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 04:59:46.825093 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 04:59:46.825190 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 04:59:46.825409 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 04:59:46.825430 (+ 21us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 04:59:46.825565 (+ 135us) client_negotiation.cc:769] Sending connection context
0924 04:59:46.825651 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 04:59:46.825809 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":73,"threads_started":1}
I20260924 04:59:46.826404 11086 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40709
I20260924 04:59:46.826484 11086 heartbeater.cc:464] Registering TS with master...
I20260924 04:59:46.826618 11086 heartbeater.cc:511] Master 127.0.98.62:40709 requested a full tablet report, sending...
I20260924 04:59:46.826918 10627 ts_manager.cc:194] Registered new tserver with Master: 8ee31b7a91ac4e88aaf53c23c9ae90b3 (127.0.98.3:42713)
I20260924 04:59:46.827240 10627 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:43135
I20260924 04:59:46.832419 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 04:59:46.838843 10685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:46.833608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45972 (local address 127.0.98.62:40709)
0924 04:59:46.833666 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:46.833668 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:46.833732 (+ 64us) server_negotiation.cc:407] Connection header received
0924 04:59:46.833792 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:46.833793 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:46.833803 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:46.833844 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 04:59:46.834374 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:46.834838 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:46.835536 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:46.835719 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:46.835772 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:46.836821 (+ 1049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 04:59:46.836823 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 04:59:46.836823 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 04:59:46.836830 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 04:59:46.837864 (+ 1034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:46.838158 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:46.838159 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:46.838160 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:46.838208 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 04:59:46.838363 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 04:59:46.838364 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 04:59:46.838364 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 04:59:46.838499 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 04:59:46.838585 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:46.838685 (+ 100us) server_negotiation.cc:299] Negotiation successful
0924 04:59:46.838756 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 04:59:46.840312 10627 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45972:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 04:59:46.841856 10627 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 04:59:46.848027 11097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:46.843896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44831 (local address 127.0.0.1:36794)
0924 04:59:46.844170 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:46.844178 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:46.844220 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:46.844585 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:46.844589 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:46.844591 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:59:46.844652 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:46.844654 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:46.845701 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:46.845702 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:46.846819 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:46.846822 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:46.847716 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:46.847717 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:46.847795 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:46.847809 (+ 14us) client_negotiation.cc:769] Sending connection context
0924 04:59:46.847895 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 04:59:46.847950 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":55,"threads_started":1}
I20260924 04:59:46.848033 11098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:46.844275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36794 (local address 127.0.98.2:44831)
0924 04:59:46.844441 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:46.844445 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:46.844453 (+ 8us) server_negotiation.cc:407] Connection header received
0924 04:59:46.844482 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:46.844483 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:46.844496 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:46.844563 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:59:46.844712 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:46.845617 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:46.846920 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:46.847621 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:46.847691 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:46.847746 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:46.847913 (+ 167us) server_negotiation.cc:299] Negotiation successful
0924 04:59:46.847952 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":73,"threads_started":1}
I20260924 04:59:46.848440 11094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:46.844079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42713 (local address 127.0.0.1:37222)
0924 04:59:46.844856 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:46.844866 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:46.844914 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:46.845125 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:46.845126 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:46.845128 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:59:46.845172 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:46.845173 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:46.846389 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:46.846390 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:46.847274 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:46.847277 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:46.848228 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:46.848229 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:46.848286 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:46.848290 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 04:59:46.848343 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 04:59:46.848391 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":765,"thread_start_us":44,"threads_started":1}
I20260924 04:59:46.848702 11095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:46.843885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40671 (local address 127.0.0.1:46556)
0924 04:59:46.844061 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:46.844102 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:46.844170 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:46.844778 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:46.844780 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:46.844782 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:59:46.844837 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:46.844839 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:46.846139 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:46.846140 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:46.847188 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:46.847190 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:46.848004 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:46.848005 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:46.848059 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:46.848063 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 04:59:46.848423 (+ 360us) client_negotiation.cc:240] Negotiation successful
0924 04:59:46.848632 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":1,"thread_start_us":68,"threads_started":1}
I20260924 04:59:46.848815 10886 tablet_service.cc:1467] Processing CreateTablet for tablet d34cd474c6274e0e891b1672e3ab10c4 (DEFAULT_TABLE table=test-table [id=0997c345b00a4f1099709c7663595d0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:59:46.848971 11096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:46.844036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46556 (local address 127.0.98.1:40671)
0924 04:59:46.844602 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:46.844606 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:46.844616 (+ 10us) server_negotiation.cc:407] Connection header received
0924 04:59:46.844657 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:46.844659 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:46.844675 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:46.844763 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:59:46.844922 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:46.846026 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:46.847273 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:46.847849 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:46.847914 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:46.847969 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:46.848827 (+ 858us) server_negotiation.cc:299] Negotiation successful
0924 04:59:46.848895 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":83,"threads_started":1}
I20260924 04:59:46.849192 10886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d34cd474c6274e0e891b1672e3ab10c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:59:46.849630 11099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:46.844425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37222 (local address 127.0.98.3:42713)
0924 04:59:46.844582 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:46.844586 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:46.844975 (+ 389us) server_negotiation.cc:407] Connection header received
0924 04:59:46.845018 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:46.845021 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:46.845037 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:46.845119 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:59:46.845232 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:46.846288 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:46.847357 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:46.848042 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:46.848109 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:46.848164 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:46.849497 (+ 1333us) server_negotiation.cc:299] Negotiation successful
0924 04:59:46.849548 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":67,"threads_started":1}
I20260924 04:59:46.849607 10752 tablet_service.cc:1467] Processing CreateTablet for tablet d34cd474c6274e0e891b1672e3ab10c4 (DEFAULT_TABLE table=test-table [id=0997c345b00a4f1099709c7663595d0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:59:46.849897 10752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d34cd474c6274e0e891b1672e3ab10c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:59:46.850250 11020 tablet_service.cc:1467] Processing CreateTablet for tablet d34cd474c6274e0e891b1672e3ab10c4 (DEFAULT_TABLE table=test-table [id=0997c345b00a4f1099709c7663595d0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 04:59:46.850584 11020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d34cd474c6274e0e891b1672e3ab10c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 04:59:46.852394 11100 tablet_bootstrap.cc:492] T d34cd474c6274e0e891b1672e3ab10c4 P 055bdbb5b4bb417f9f338d61d056def2: Bootstrap starting.
I20260924 04:59:46.852798 11101 tablet_bootstrap.cc:492] T d34cd474c6274e0e891b1672e3ab10c4 P d2c39ba5a76740229afa6c481ac27d5a: Bootstrap starting.
I20260924 04:59:46.852994 11100 tablet_bootstrap.cc:654] T d34cd474c6274e0e891b1672e3ab10c4 P 055bdbb5b4bb417f9f338d61d056def2: Neither blocks nor log segments found. Creating new log.
I20260924 04:59:46.853440 11101 tablet_bootstrap.cc:654] T d34cd474c6274e0e891b1672e3ab10c4 P d2c39ba5a76740229afa6c481ac27d5a: Neither blocks nor log segments found. Creating new log.
I20260924 04:59:46.853509 11100 log.cc:824] T d34cd474c6274e0e891b1672e3ab10c4 P 055bdbb5b4bb417f9f338d61d056def2: Log is configured to *not* fsync() on all Append() calls
I20260924 04:59:46.853816 11102 tablet_bootstrap.cc:492] T d34cd474c6274e0e891b1672e3ab10c4 P 8ee31b7a91ac4e88aaf53c23c9ae90b3: Bootstrap starting.
I20260924 04:59:46.854008 11101 log.cc:824] T d34cd474c6274e0e891b1672e3ab10c4 P d2c39ba5a76740229afa6c481ac27d5a: Log is configured to *not* fsync() on all Append() calls
I20260924 04:59:46.854405 11102 tablet_bootstrap.cc:654] T d34cd474c6274e0e891b1672e3ab10c4 P 8ee31b7a91ac4e88aaf53c23c9ae90b3: Neither blocks nor log segments found. Creating new log.
I20260924 04:59:46.854813 11101 tablet_bootstrap.cc:492] T d34cd474c6274e0e891b1672e3ab10c4 P d2c39ba5a76740229afa6c481ac27d5a: No bootstrap required, opened a new log
I20260924 04:59:46.854892 11101 ts_tablet_manager.cc:1397] T d34cd474c6274e0e891b1672e3ab10c4 P d2c39ba5a76740229afa6c481ac27d5a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 04:59:46.855261 11102 log.cc:824] T d34cd474c6274e0e891b1672e3ab10c4 P 8ee31b7a91ac4e88aaf53c23c9ae90b3: Log is configured to *not* fsync() on all Append() calls
I20260924 04:59:46.855556 11100 tablet_bootstrap.cc:492] T d34cd474c6274e0e891b1672e3ab10c4 P 055bdbb5b4bb417f9f338d61d056def2: No bootstrap required, opened a new log
I20260924 04:59:46.855623 11100 ts_tablet_manager.cc:1397] T d34cd474c6274e0e891b1672e3ab10c4 P 055bdbb5b4bb417f9f338d61d056def2: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260924 04:59:46.856137 11102 tablet_bootstrap.cc:492] T d34cd474c6274e0e891b1672e3ab10c4 P 8ee31b7a91ac4e88aaf53c23c9ae90b3: No bootstrap required, opened a new log
I20260924 04:59:46.856220 11102 ts_tablet_manager.cc:1397] T d34cd474c6274e0e891b1672e3ab10c4 P 8ee31b7a91ac4e88aaf53c23c9ae90b3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 04:59:46.856796 11101 raft_consensus.cc:348] T d34cd474c6274e0e891b1672e3ab10c4 P d2c39ba5a76740229afa6c481ac27d5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "055bdbb5b4bb417f9f338d61d056def2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40671 } } peers { permanent_uuid: "d2c39ba5a76740229afa6c481ac27d5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44831 } } peers { permanent_uuid: "8ee31b7a91ac4e88aaf53c23c9ae90b3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42713 } }
I20260924 04:59:46.856982 11101 raft_consensus.cc:374] T d34cd474c6274e0e891b1672e3ab10c4 P d2c39ba5a76740229afa6c481ac27d5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:59:46.857021 11101 raft_consensus.cc:729] T d34cd474c6274e0e891b1672e3ab10c4 P d2c39ba5a76740229afa6c481ac27d5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2c39ba5a76740229afa6c481ac27d5a, State: Initialized, Role: FOLLOWER
I20260924 04:59:46.857069 11100 raft_consensus.cc:348] T d34cd474c6274e0e891b1672e3ab10c4 P 055bdbb5b4bb417f9f338d61d056def2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "055bdbb5b4bb417f9f338d61d056def2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40671 } } peers { permanent_uuid: "d2c39ba5a76740229afa6c481ac27d5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44831 } } peers { permanent_uuid: "8ee31b7a91ac4e88aaf53c23c9ae90b3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42713 } }
I20260924 04:59:46.857161 11101 consensus_queue.cc:260] T d34cd474c6274e0e891b1672e3ab10c4 P d2c39ba5a76740229afa6c481ac27d5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "055bdbb5b4bb417f9f338d61d056def2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40671 } } peers { permanent_uuid: "d2c39ba5a76740229afa6c481ac27d5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44831 } } peers { permanent_uuid: "8ee31b7a91ac4e88aaf53c23c9ae90b3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42713 } }
I20260924 04:59:46.857225 11100 raft_consensus.cc:374] T d34cd474c6274e0e891b1672e3ab10c4 P 055bdbb5b4bb417f9f338d61d056def2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:59:46.857264 11100 raft_consensus.cc:729] T d34cd474c6274e0e891b1672e3ab10c4 P 055bdbb5b4bb417f9f338d61d056def2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 055bdbb5b4bb417f9f338d61d056def2, State: Initialized, Role: FOLLOWER
I20260924 04:59:46.857364 11100 consensus_queue.cc:260] T d34cd474c6274e0e891b1672e3ab10c4 P 055bdbb5b4bb417f9f338d61d056def2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "055bdbb5b4bb417f9f338d61d056def2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40671 } } peers { permanent_uuid: "d2c39ba5a76740229afa6c481ac27d5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44831 } } peers { permanent_uuid: "8ee31b7a91ac4e88aaf53c23c9ae90b3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42713 } }
I20260924 04:59:46.857463 11101 ts_tablet_manager.cc:1428] T d34cd474c6274e0e891b1672e3ab10c4 P d2c39ba5a76740229afa6c481ac27d5a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 04:59:46.857632 11100 ts_tablet_manager.cc:1428] T d34cd474c6274e0e891b1672e3ab10c4 P 055bdbb5b4bb417f9f338d61d056def2: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260924 04:59:46.857757 10952 heartbeater.cc:503] Master 127.0.98.62:40709 was elected leader, sending a full tablet report...
I20260924 04:59:46.857970 10818 heartbeater.cc:503] Master 127.0.98.62:40709 was elected leader, sending a full tablet report...
I20260924 04:59:46.858162 11102 raft_consensus.cc:348] T d34cd474c6274e0e891b1672e3ab10c4 P 8ee31b7a91ac4e88aaf53c23c9ae90b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "055bdbb5b4bb417f9f338d61d056def2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40671 } } peers { permanent_uuid: "d2c39ba5a76740229afa6c481ac27d5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44831 } } peers { permanent_uuid: "8ee31b7a91ac4e88aaf53c23c9ae90b3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42713 } }
I20260924 04:59:46.858351 11102 raft_consensus.cc:374] T d34cd474c6274e0e891b1672e3ab10c4 P 8ee31b7a91ac4e88aaf53c23c9ae90b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 04:59:46.858388 11102 raft_consensus.cc:729] T d34cd474c6274e0e891b1672e3ab10c4 P 8ee31b7a91ac4e88aaf53c23c9ae90b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ee31b7a91ac4e88aaf53c23c9ae90b3, State: Initialized, Role: FOLLOWER
I20260924 04:59:46.858498 11102 consensus_queue.cc:260] T d34cd474c6274e0e891b1672e3ab10c4 P 8ee31b7a91ac4e88aaf53c23c9ae90b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "055bdbb5b4bb417f9f338d61d056def2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40671 } } peers { permanent_uuid: "d2c39ba5a76740229afa6c481ac27d5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44831 } } peers { permanent_uuid: "8ee31b7a91ac4e88aaf53c23c9ae90b3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42713 } }
I20260924 04:59:46.858747 11102 ts_tablet_manager.cc:1428] T d34cd474c6274e0e891b1672e3ab10c4 P 8ee31b7a91ac4e88aaf53c23c9ae90b3: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 04:59:46.858796 11086 heartbeater.cc:503] Master 127.0.98.62:40709 was elected leader, sending a full tablet report...
W20260924 04:59:46.922889 10953 tablet.cc:2369] T d34cd474c6274e0e891b1672e3ab10c4 P d2c39ba5a76740229afa6c481ac27d5a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:59:46.983057 11108 raft_consensus.cc:482] T d34cd474c6274e0e891b1672e3ab10c4 P 8ee31b7a91ac4e88aaf53c23c9ae90b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 04:59:46.983194 11108 raft_consensus.cc:504] T d34cd474c6274e0e891b1672e3ab10c4 P 8ee31b7a91ac4e88aaf53c23c9ae90b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "055bdbb5b4bb417f9f338d61d056def2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40671 } } peers { permanent_uuid: "d2c39ba5a76740229afa6c481ac27d5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44831 } } peers { permanent_uuid: "8ee31b7a91ac4e88aaf53c23c9ae90b3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42713 } }
I20260924 04:59:46.983530 11108 leader_election.cc:288] T d34cd474c6274e0e891b1672e3ab10c4 P 8ee31b7a91ac4e88aaf53c23c9ae90b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 055bdbb5b4bb417f9f338d61d056def2 (127.0.98.1:40671), d2c39ba5a76740229afa6c481ac27d5a (127.0.98.2:44831)
I20260924 04:59:46.986346 11088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:46.983717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40671 (local address 127.0.98.3:43347)
0924 04:59:46.983768 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:46.983780 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:46.983824 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:46.983952 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:46.983952 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:46.983954 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:59:46.984025 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:46.984027 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:46.984668 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:46.984669 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:46.985410 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:46.985412 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:46.986131 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:46.986132 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:46.986185 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:46.986188 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 04:59:46.986219 (+ 31us) client_negotiation.cc:240] Negotiation successful
0924 04:59:46.986248 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260924 04:59:46.986341 11096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:46.983786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43347 (local address 127.0.98.1:40671)
0924 04:59:46.983826 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:46.983827 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:46.983836 (+ 9us) server_negotiation.cc:407] Connection header received
0924 04:59:46.983884 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:46.983885 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:46.983895 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:46.983943 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:59:46.984086 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:46.984593 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:46.985526 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:46.986064 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:46.986135 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:46.986145 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:46.986239 (+ 94us) server_negotiation.cc:299] Negotiation successful
0924 04:59:46.986272 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 04:59:46.986572 11098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:46.983782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44981 (local address 127.0.98.2:44831)
0924 04:59:46.983836 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:46.983838 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:46.984054 (+ 216us) server_negotiation.cc:407] Connection header received
0924 04:59:46.984095 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:46.984096 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:46.984107 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:46.984150 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 04:59:46.984322 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:46.984778 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:46.985701 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:46.986242 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:46.986302 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:46.986311 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:46.986465 (+ 154us) server_negotiation.cc:299] Negotiation successful
0924 04:59:46.986510 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 04:59:46.986711 10772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d34cd474c6274e0e891b1672e3ab10c4" candidate_uuid: "8ee31b7a91ac4e88aaf53c23c9ae90b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "055bdbb5b4bb417f9f338d61d056def2" is_pre_election: true
I20260924 04:59:46.986776 11109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:46.983718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44831 (local address 127.0.98.3:44981)
0924 04:59:46.983977 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:46.983986 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 04:59:46.984043 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 04:59:46.984161 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 04:59:46.984162 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 04:59:46.984163 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 04:59:46.984229 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:46.984231 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:46.984849 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:46.984850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:46.985631 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 04:59:46.985634 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 04:59:46.986312 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 04:59:46.986313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 04:59:46.986373 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:46.986377 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 04:59:46.986655 (+ 278us) client_negotiation.cc:240] Negotiation successful
0924 04:59:46.986692 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"mutex_wait_us":73,"thread_start_us":122,"threads_started":1}
I20260924 04:59:46.986873 10772 raft_consensus.cc:2454] T d34cd474c6274e0e891b1672e3ab10c4 P 055bdbb5b4bb417f9f338d61d056def2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ee31b7a91ac4e88aaf53c23c9ae90b3 in term 0.
I20260924 04:59:46.987095 10974 leader_election.cc:302] T d34cd474c6274e0e891b1672e3ab10c4 P 8ee31b7a91ac4e88aaf53c23c9ae90b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 055bdbb5b4bb417f9f338d61d056def2, 8ee31b7a91ac4e88aaf53c23c9ae90b3; no voters:
I20260924 04:59:46.987097 10906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d34cd474c6274e0e891b1672e3ab10c4" candidate_uuid: "8ee31b7a91ac4e88aaf53c23c9ae90b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2c39ba5a76740229afa6c481ac27d5a" is_pre_election: true
I20260924 04:59:46.987214 10906 raft_consensus.cc:2454] T d34cd474c6274e0e891b1672e3ab10c4 P d2c39ba5a76740229afa6c481ac27d5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ee31b7a91ac4e88aaf53c23c9ae90b3 in term 0.
I20260924 04:59:46.987221 11108 raft_consensus.cc:2781] T d34cd474c6274e0e891b1672e3ab10c4 P 8ee31b7a91ac4e88aaf53c23c9ae90b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 04:59:46.987272 11108 raft_consensus.cc:482] T d34cd474c6274e0e891b1672e3ab10c4 P 8ee31b7a91ac4e88aaf53c23c9ae90b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 04:59:46.987299 11108 raft_consensus.cc:3037] T d34cd474c6274e0e891b1672e3ab10c4 P 8ee31b7a91ac4e88aaf53c23c9ae90b3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:59:46.987990 11108 raft_consensus.cc:504] T d34cd474c6274e0e891b1672e3ab10c4 P 8ee31b7a91ac4e88aaf53c23c9ae90b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "055bdbb5b4bb417f9f338d61d056def2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40671 } } peers { permanent_uuid: "d2c39ba5a76740229afa6c481ac27d5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44831 } } peers { permanent_uuid: "8ee31b7a91ac4e88aaf53c23c9ae90b3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42713 } }
I20260924 04:59:46.988106 11108 leader_election.cc:288] T d34cd474c6274e0e891b1672e3ab10c4 P 8ee31b7a91ac4e88aaf53c23c9ae90b3 [CANDIDATE]: Term 1 election: Requested vote from peers 055bdbb5b4bb417f9f338d61d056def2 (127.0.98.1:40671), d2c39ba5a76740229afa6c481ac27d5a (127.0.98.2:44831)
I20260924 04:59:46.988247 10772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d34cd474c6274e0e891b1672e3ab10c4" candidate_uuid: "8ee31b7a91ac4e88aaf53c23c9ae90b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "055bdbb5b4bb417f9f338d61d056def2"
I20260924 04:59:46.988293 10906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d34cd474c6274e0e891b1672e3ab10c4" candidate_uuid: "8ee31b7a91ac4e88aaf53c23c9ae90b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2c39ba5a76740229afa6c481ac27d5a"
I20260924 04:59:46.988322 10772 raft_consensus.cc:3037] T d34cd474c6274e0e891b1672e3ab10c4 P 055bdbb5b4bb417f9f338d61d056def2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:59:46.988366 10906 raft_consensus.cc:3037] T d34cd474c6274e0e891b1672e3ab10c4 P d2c39ba5a76740229afa6c481ac27d5a [term 0 FOLLOWER]: Advancing to term 1
I20260924 04:59:46.989071 10772 raft_consensus.cc:2454] T d34cd474c6274e0e891b1672e3ab10c4 P 055bdbb5b4bb417f9f338d61d056def2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ee31b7a91ac4e88aaf53c23c9ae90b3 in term 1.
I20260924 04:59:46.989084 10906 raft_consensus.cc:2454] T d34cd474c6274e0e891b1672e3ab10c4 P d2c39ba5a76740229afa6c481ac27d5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ee31b7a91ac4e88aaf53c23c9ae90b3 in term 1.
I20260924 04:59:46.989261 10973 leader_election.cc:302] T d34cd474c6274e0e891b1672e3ab10c4 P 8ee31b7a91ac4e88aaf53c23c9ae90b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ee31b7a91ac4e88aaf53c23c9ae90b3, d2c39ba5a76740229afa6c481ac27d5a; no voters:
I20260924 04:59:46.989373 11108 raft_consensus.cc:2781] T d34cd474c6274e0e891b1672e3ab10c4 P 8ee31b7a91ac4e88aaf53c23c9ae90b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 04:59:46.989543 11108 raft_consensus.cc:686] T d34cd474c6274e0e891b1672e3ab10c4 P 8ee31b7a91ac4e88aaf53c23c9ae90b3 [term 1 LEADER]: Becoming Leader. State: Replica: 8ee31b7a91ac4e88aaf53c23c9ae90b3, State: Running, Role: LEADER
I20260924 04:59:46.989638 11108 consensus_queue.cc:237] T d34cd474c6274e0e891b1672e3ab10c4 P 8ee31b7a91ac4e88aaf53c23c9ae90b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "055bdbb5b4bb417f9f338d61d056def2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40671 } } peers { permanent_uuid: "d2c39ba5a76740229afa6c481ac27d5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44831 } } peers { permanent_uuid: "8ee31b7a91ac4e88aaf53c23c9ae90b3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42713 } }
I20260924 04:59:46.990486 10626 catalog_manager.cc:5495] T d34cd474c6274e0e891b1672e3ab10c4 P 8ee31b7a91ac4e88aaf53c23c9ae90b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ee31b7a91ac4e88aaf53c23c9ae90b3 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "8ee31b7a91ac4e88aaf53c23c9ae90b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "055bdbb5b4bb417f9f338d61d056def2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d2c39ba5a76740229afa6c481ac27d5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ee31b7a91ac4e88aaf53c23c9ae90b3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42713 } health_report { overall_health: HEALTHY } } }
W20260924 04:59:47.026401 10819 tablet.cc:2369] T d34cd474c6274e0e891b1672e3ab10c4 P 055bdbb5b4bb417f9f338d61d056def2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 04:59:47.031297 11099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:47.028963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37226 (local address 127.0.98.3:42713)
0924 04:59:47.029004 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 04:59:47.029020 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 04:59:47.029028 (+ 8us) server_negotiation.cc:407] Connection header received
0924 04:59:47.029063 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 04:59:47.029064 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 04:59:47.029083 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 04:59:47.029140 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 04:59:47.029284 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:47.029915 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:47.030662 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 04:59:47.030850 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 04:59:47.030899 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 04:59:47.031021 (+ 122us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 04:59:47.031117 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 04:59:47.031169 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 04:59:47.031197 (+ 28us) server_negotiation.cc:299] Negotiation successful
0924 04:59:47.031215 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 04:59:47.033078 10772 raft_consensus.cc:1261] T d34cd474c6274e0e891b1672e3ab10c4 P 055bdbb5b4bb417f9f338d61d056def2 [term 1 FOLLOWER]: Refusing update from remote peer 8ee31b7a91ac4e88aaf53c23c9ae90b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:59:47.033111 10906 raft_consensus.cc:1261] T d34cd474c6274e0e891b1672e3ab10c4 P d2c39ba5a76740229afa6c481ac27d5a [term 1 FOLLOWER]: Refusing update from remote peer 8ee31b7a91ac4e88aaf53c23c9ae90b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 04:59:47.033408 11110 consensus_queue.cc:1035] T d34cd474c6274e0e891b1672e3ab10c4 P 8ee31b7a91ac4e88aaf53c23c9ae90b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "055bdbb5b4bb417f9f338d61d056def2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:59:47.033557 11108 consensus_queue.cc:1035] T d34cd474c6274e0e891b1672e3ab10c4 P 8ee31b7a91ac4e88aaf53c23c9ae90b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2c39ba5a76740229afa6c481ac27d5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 04:59:47.035817 11114 mvcc.cc:205] Tried to move back new op lower bound from 7332765642884816896 to 7332765642710241280. Current Snapshot: MvccSnapshot[applied={T|T < 7332765642884816896}]
I20260924 04:59:47.038080 10626 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45972:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:59:47.038179 10626 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45972:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 04:59:47.038877 10626 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 16a60adc553b438b8b26acfa4f84ce1c: Sending DeleteTablet for 3 replicas of tablet d34cd474c6274e0e891b1672e3ab10c4
I20260924 04:59:47.039223 11020 tablet_service.cc:1514] Processing DeleteTablet for tablet d34cd474c6274e0e891b1672e3ab10c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:59:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37222
I20260924 04:59:47.039223 10752 tablet_service.cc:1514] Processing DeleteTablet for tablet d34cd474c6274e0e891b1672e3ab10c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:59:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46556
I20260924 04:59:47.039237 10886 tablet_service.cc:1514] Processing DeleteTablet for tablet d34cd474c6274e0e891b1672e3ab10c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 04:59:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36794
I20260924 04:59:47.039438 11123 tablet_replica.cc:330] T d34cd474c6274e0e891b1672e3ab10c4 P 055bdbb5b4bb417f9f338d61d056def2: stopping tablet replica
I20260924 04:59:47.039534 11123 raft_consensus.cc:2229] T d34cd474c6274e0e891b1672e3ab10c4 P 055bdbb5b4bb417f9f338d61d056def2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:59:47.039625 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 10689
I20260924 04:59:47.039640 11123 raft_consensus.cc:2258] T d34cd474c6274e0e891b1672e3ab10c4 P 055bdbb5b4bb417f9f338d61d056def2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:59:47.040685 11125 tablet_replica.cc:330] T d34cd474c6274e0e891b1672e3ab10c4 P d2c39ba5a76740229afa6c481ac27d5a: stopping tablet replica
I20260924 04:59:47.040784 11125 raft_consensus.cc:2229] T d34cd474c6274e0e891b1672e3ab10c4 P d2c39ba5a76740229afa6c481ac27d5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 04:59:47.040884 11125 raft_consensus.cc:2258] T d34cd474c6274e0e891b1672e3ab10c4 P d2c39ba5a76740229afa6c481ac27d5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 04:59:47.041410 11125 ts_tablet_manager.cc:1905] T d34cd474c6274e0e891b1672e3ab10c4 P d2c39ba5a76740229afa6c481ac27d5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:59:47.043165 11125 ts_tablet_manager.cc:1918] T d34cd474c6274e0e891b1672e3ab10c4 P d2c39ba5a76740229afa6c481ac27d5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:59:47.043286 11125 log.cc:1196] T d34cd474c6274e0e891b1672e3ab10c4 P d2c39ba5a76740229afa6c481ac27d5a: Deleting WAL directory at /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225929848825-392-0/minicluster-data/ts-1/wal/wals/d34cd474c6274e0e891b1672e3ab10c4
I20260924 04:59:47.043514 11125 ts_tablet_manager.cc:1939] T d34cd474c6274e0e891b1672e3ab10c4 P d2c39ba5a76740229afa6c481ac27d5a: Deleting consensus metadata
I20260924 04:59:47.043866 10611 catalog_manager.cc:4841] TS d2c39ba5a76740229afa6c481ac27d5a (127.0.98.2:44831): tablet d34cd474c6274e0e891b1672e3ab10c4 (table test-table [id=0997c345b00a4f1099709c7663595d0a]) successfully deleted
I20260924 04:59:47.045542 11124 tablet_replica.cc:330] T d34cd474c6274e0e891b1672e3ab10c4 P 8ee31b7a91ac4e88aaf53c23c9ae90b3: stopping tablet replica
I20260924 04:59:47.045650 11124 raft_consensus.cc:2229] T d34cd474c6274e0e891b1672e3ab10c4 P 8ee31b7a91ac4e88aaf53c23c9ae90b3 [term 1 LEADER]: Raft consensus shutting down.
I20260924 04:59:47.045792 11124 raft_consensus.cc:2258] T d34cd474c6274e0e891b1672e3ab10c4 P 8ee31b7a91ac4e88aaf53c23c9ae90b3 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260924 04:59:47.045802 10612 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40671 (error 108)
I20260924 04:59:47.046171 11124 ts_tablet_manager.cc:1905] T d34cd474c6274e0e891b1672e3ab10c4 P 8ee31b7a91ac4e88aaf53c23c9ae90b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 04:59:47.046521 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 10823
I20260924 04:59:47.046537 11095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 04:59:47.046401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40671 (local address 127.0.0.1:46570)
0924 04:59:47.046458 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0924 04:59:47.046503 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260924 04:59:47.047223 10612 catalog_manager.cc:4584] TS 055bdbb5b4bb417f9f338d61d056def2 (127.0.98.1:40671): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d34cd474c6274e0e891b1672e3ab10c4: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40671: connect: Connection refused (error 111)
I20260924 04:59:47.048316 11124 ts_tablet_manager.cc:1918] T d34cd474c6274e0e891b1672e3ab10c4 P 8ee31b7a91ac4e88aaf53c23c9ae90b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 04:59:47.048384 11124 log.cc:1196] T d34cd474c6274e0e891b1672e3ab10c4 P 8ee31b7a91ac4e88aaf53c23c9ae90b3: Deleting WAL directory at /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790225929848825-392-0/minicluster-data/ts-2/wal/wals/d34cd474c6274e0e891b1672e3ab10c4
I20260924 04:59:47.048599 11124 ts_tablet_manager.cc:1939] T d34cd474c6274e0e891b1672e3ab10c4 P 8ee31b7a91ac4e88aaf53c23c9ae90b3: Deleting consensus metadata
I20260924 04:59:47.048946 10612 catalog_manager.cc:4841] TS 8ee31b7a91ac4e88aaf53c23c9ae90b3 (127.0.98.3:42713): tablet d34cd474c6274e0e891b1672e3ab10c4 (table test-table [id=0997c345b00a4f1099709c7663595d0a]) successfully deleted
I20260924 04:59:47.053033 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 10957
I20260924 04:59:47.057843 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 10595
2026-09-24T04:59:47Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (1927 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 04:59:47 dist-test-slave-70st krb5kdc[11129](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 04:59:47 dist-test-slave-70st krb5kdc[11129](info): set up 2 sockets
Sep 24 04:59:47 dist-test-slave-70st krb5kdc[11129](info): commencing operation
krb5kdc: starting...
W20260924 04:59:48.006111 392 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/krb5kdc/test-user.keytab.
Sep 24 04:59:48 dist-test-slave-70st krb5kdc[11129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790225988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T04:59:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T04:59:48Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260924 04:59:48.160121 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskZbcc3B/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/minicluster-data/postgres -l logfile start
2026-09-24 04:59:50.486 UTC [11159] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 04:59:50.486 UTC [11159] LOG: listening on IPv4 address "127.0.98.20", port 49705
2026-09-24 04:59:50.490 UTC [11159] LOG: listening on Unix socket "/tmp/.s.PGSQL.49705"
2026-09-24 04:59:50.515 UTC [11162] LOG: database system was shut down at 2026-09-24 04:59:49 UTC
2026-09-24 04:59:50.519 UTC [11159] LOG: database system is ready to accept connections
I20260924 04:59:51.381894 392 mini_postgres.cc:96] Postgres bound to 49705
2026-09-24 04:59:51.387 UTC [11170] FATAL: database "slave" does not exist
127.0.98.20:49705 - accepting connections
I20260924 04:59:51.387985 392 mini_ranger.cc:161] Starting Ranger...
I20260924 04:59:51.401520 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 04:59:51.849310 392 mini_ranger.cc:87] Created ranger Postgres database
I20260924 04:59:51.849380 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/minicluster-data/ranger-admin
2026-09-24 04:59:52,125 [I] DB FLAVOR :POSTGRES
2026-09-24 04:59:52,126 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 04:59:52,126 [I] Checking connection..
2026-09-24 04:59:52,126 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZbcc3B/build/release/bin/postgresql.jar:/tmp/dist-test-taskZbcc3B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49705/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 04:59:52,363 [I] Checking connection passed.
2026-09-24 04:59:52,363 [I] --------- Verifying version history table ---------
2026-09-24 04:59:52,363 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZbcc3B/build/release/bin/postgresql.jar:/tmp/dist-test-taskZbcc3B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49705/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 04:59:52,617 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 04:59:52,617 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZbcc3B/build/release/bin/postgresql.jar:/tmp/dist-test-taskZbcc3B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49705/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 04:59:52,855 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 04:59:52,856 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 04:59:52,856 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZbcc3B/build/release/bin/postgresql.jar:/tmp/dist-test-taskZbcc3B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49705/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskZbcc3B/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 04:59:53.097 UTC [11234] WARNING: there is no transaction in progress
2026-09-24 04:59:53,102 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 04:59:53,103 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZbcc3B/build/release/bin/postgresql.jar:/tmp/dist-test-taskZbcc3B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49705/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 04:59:53,337 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 04:59:53,337 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 04:59:53,338 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZbcc3B/build/release/bin/postgresql.jar:/tmp/dist-test-taskZbcc3B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49705/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 04:59:53,569 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZbcc3B/build/release/bin/postgresql.jar:/tmp/dist-test-taskZbcc3B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49705/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 04:59:53,815 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZbcc3B/build/release/bin/postgresql.jar:/tmp/dist-test-taskZbcc3B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49705/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-70st.c.gcp-upstream.internal','N') ;"
2026-09-24 04:59:54,050 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZbcc3B/build/release/bin/postgresql.jar:/tmp/dist-test-taskZbcc3B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49705/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 04:59:54,276 [I] Table x_portal_user does not exist in database ranger
2026-09-24 04:59:54,276 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZbcc3B/build/release/bin/postgresql.jar:/tmp/dist-test-taskZbcc3B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49705/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 04:59:54,508 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 04:59:54,509 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZbcc3B/build/release/bin/postgresql.jar:/tmp/dist-test-taskZbcc3B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49705/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 04:59:54,740 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZbcc3B/build/release/bin/postgresql.jar:/tmp/dist-test-taskZbcc3B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49705/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 04:59:55,000 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 04:59:55,001 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZbcc3B/build/release/bin/postgresql.jar:/tmp/dist-test-taskZbcc3B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49705/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskZbcc3B/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 04:59:55.904 UTC [11342] WARNING: there is no transaction in progress
2026-09-24 04:59:55.919 UTC [11342] WARNING: there is no transaction in progress
2026-09-24 04:59:55.935 UTC [11342] WARNING: there is no transaction in progress
2026-09-24 04:59:55.950 UTC [11342] WARNING: there is no transaction in progress
2026-09-24 04:59:55.965 UTC [11342] WARNING: there is no transaction in progress
2026-09-24 04:59:55.982 UTC [11342] WARNING: there is no transaction in progress
2026-09-24 04:59:56.039 UTC [11342] WARNING: there is no transaction in progress
2026-09-24 04:59:56.046 UTC [11342] WARNING: there is no transaction in progress
2026-09-24 04:59:56.059 UTC [11342] WARNING: there is no transaction in progress
2026-09-24 04:59:56.069 UTC [11342] WARNING: there is no transaction in progress
2026-09-24 04:59:56.078 UTC [11342] WARNING: there is no transaction in progress
2026-09-24 04:59:56.092 UTC [11342] WARNING: there is no transaction in progress
2026-09-24 04:59:56.101 UTC [11342] WARNING: there is no transaction in progress
2026-09-24 04:59:56.110 UTC [11342] WARNING: there is no transaction in progress
2026-09-24 04:59:56.118 UTC [11342] WARNING: there is no transaction in progress
2026-09-24 04:59:56.125 UTC [11342] WARNING: there is no transaction in progress
2026-09-24 04:59:56,735 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 04:59:56,736 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZbcc3B/build/release/bin/postgresql.jar:/tmp/dist-test-taskZbcc3B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49705/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 04:59:56,980 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 04:59:56,981 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZbcc3B/build/release/bin/postgresql.jar:/tmp/dist-test-taskZbcc3B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49705/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 04:59:57,220 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 04:59:57,221 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZbcc3B/build/release/bin/postgresql.jar:/tmp/dist-test-taskZbcc3B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49705/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 04:59:57,473 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 04:59:57,473 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZbcc3B/build/release/bin/postgresql.jar:/tmp/dist-test-taskZbcc3B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49705/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 04:59:57,725 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZbcc3B/build/release/bin/postgresql.jar:/tmp/dist-test-taskZbcc3B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49705/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 04:59:57,977 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZbcc3B/build/release/bin/postgresql.jar:/tmp/dist-test-taskZbcc3B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49705/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-70st.c.gcp-upstream.internal';"
2026-09-24 04:59:58,224 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 04:59:58,225 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZbcc3B/build/release/bin/postgresql.jar:/tmp/dist-test-taskZbcc3B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49705/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 04:59:58,477 [I] DB_PATCHES have already been applied
I20260924 04:59:58.483163 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskZbcc3B/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskZbcc3B/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskZbcc3B/build/release/bin/postgresql.jar:/tmp/dist-test-taskZbcc3B/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 04:59:58.483230 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260924 04:59:58.485989 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:40887
Sep 24, 2026 4:59:58 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 4:59:58 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 4:59:58 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 4:59:58 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 4:59:58 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 4:59:59 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 4:59:59 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 04:59:59.480407 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.092s user 0.003s sys 0.005s
Sep 24, 2026 5:00:19 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 5:00:19 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 24, 2026 5:00:19 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 5:00:19 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 5:00:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 5:00:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 5:00:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 5:00:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 5:00:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 5:00:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 5:00:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 5:00:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 5:00:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 5:00:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 5:00:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 5:00:19 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 5:00:20 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 5:00:20 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 05:00:21.570653 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 05:00:21 dist-test-slave-70st krb5kdc[11129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226021, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 05:00:21.657701 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 05:00:21.667304 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 05:00:22.521044 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 05:00:22.521128 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/minicluster-data/ranger-kms
I20260924 05:00:22.521143 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:49705
2026-09-24 05:00:22,686 [I] DB FLAVOR :POSTGRES
2026-09-24 05:00:22,686 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 05:00:22,686 [I] Checking connection
2026-09-24 05:00:22,687 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZbcc3B/build/release/bin/postgresql.jar:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49705/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 05:00:22,930 [I] connection success
2026-09-24 05:00:22,931 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 05:00:22,931 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 05:00:22,931 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZbcc3B/build/release/bin/postgresql.jar:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49705/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 05:00:23,165 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 05:00:23,166 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 05:00:23,166 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 05:00:23,166 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZbcc3B/build/release/bin/postgresql.jar:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49705/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 05:00:23,411 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 05:00:23.858856 392 mini_ranger_kms.cc:310] Created kms service
I20260924 05:00:24.049363 392 mini_ranger_kms.cc:326] Created kudu user
I20260924 05:00:24.091512 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 05:00:24 dist-test-slave-70st krb5kdc[11129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226021, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 05:00:24.462728 392 mini_ranger_kms.cc:384] Added ranger policy
I20260924 05:00:24.462841 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskZbcc3B/build/release/bin/postgresql.jar:/tmp/dist-test-taskZbcc3B/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskZbcc3B/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 05:00:24.462862 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260924 05:00:24.466252 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11522
I20260924 05:00:24.466351 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:46783
Sep 24, 2026 5:00:25 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 5:00:25 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 5:00:25 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 5:00:25 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 5:00:26 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 5:00:26 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260924 05:00:26.575237 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.918s user 0.003s sys 0.145s
I20260924 05:00:26.575356 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 05:00:26.575438 392 mini_ranger_kms.cc:401] 127.0.98.20:46783/kms/v1/keys
Sep 24 05:00:26 dist-test-slave-70st krb5kdc[11129](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790226026, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 24 05:00:27 dist-test-slave-70st krb5kdc[11129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226026, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 24 05:00:27 dist-test-slave-70st krb5kdc[11129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226026, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 24 05:00:29 dist-test-slave-70st krb5kdc[11129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226029, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 05:00:29.839679 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39797
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39797
--ranger_config_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:46783/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:00:29.917824 11573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:00:29.918048 11573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:00:29.918077 11573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:00:29.919204 11573 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:00:29.919243 11573 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:00:29.919261 11573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:00:29.919277 11573 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:00:29.919292 11573 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:00:29.920783 11573 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:46783/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39797
--ranger_config_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39797
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11573
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:00:29.921008 11573 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:00:29.921200 11573 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:00:29.925808 11578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:00:29.925935 11579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:00:29.925808 11581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:00:29.926169 11573 server_base.cc:1034] running on GCE node
I20260924 05:00:29.926432 11573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:00:29.926786 11573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:00:29.927975 11573 hybrid_clock.cc:648] HybridClock initialized: now 1790226029927949 us; error 37 us; skew 500 ppm
Sep 24 05:00:29 dist-test-slave-70st krb5kdc[11129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226029, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:00:29.931258 11573 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 05:00:29.931766 11573 webserver.cc:460] Webserver started at http://127.0.98.62:40335/ using document root <none> and password file <none>
I20260924 05:00:29.932018 11573 fs_manager.cc:359] Metadata directory not provided
I20260924 05:00:29.932073 11573 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:00:29.932202 11573 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 05:00:29 dist-test-slave-70st krb5kdc[11129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226029, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 05:00:29.983352 11573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/minicluster-data/master-0/data/instance:
uuid: "8389a786572b4aaabc19b179ed4d20fc"
format_stamp: "Formatted at 2026-09-24 05:00:29 on dist-test-slave-70st"
server_key: "fe83edb403001dbd413571094f1cc647"
server_key_iv: "2d7f5df952ee15f111f5db7ab649589b"
server_key_version: "kuduclusterkey@0"
I20260924 05:00:29.983776 11573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/minicluster-data/master-0/wal/instance:
uuid: "8389a786572b4aaabc19b179ed4d20fc"
format_stamp: "Formatted at 2026-09-24 05:00:29 on dist-test-slave-70st"
server_key: "fe83edb403001dbd413571094f1cc647"
server_key_iv: "2d7f5df952ee15f111f5db7ab649589b"
server_key_version: "kuduclusterkey@0"
I20260924 05:00:29.985530 11573 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260924 05:00:29.999615 11588 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:00:29.999943 11573 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 05:00:30.000046 11573 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/minicluster-data/master-0/wal
uuid: "8389a786572b4aaabc19b179ed4d20fc"
format_stamp: "Formatted at 2026-09-24 05:00:29 on dist-test-slave-70st"
server_key: "fe83edb403001dbd413571094f1cc647"
server_key_iv: "2d7f5df952ee15f111f5db7ab649589b"
server_key_version: "kuduclusterkey@0"
I20260924 05:00:30.000141 11573 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:00:30.039702 11573 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:00:30.041654 11573 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:00:30.041774 11573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:00:30.046494 11573 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39797
I20260924 05:00:30.046528 11640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39797 every 8 connection(s)
I20260924 05:00:30.046900 11573 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:00:30.047633 11641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:00:30.050623 11641 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8389a786572b4aaabc19b179ed4d20fc: Bootstrap starting.
I20260924 05:00:30.051342 11641 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8389a786572b4aaabc19b179ed4d20fc: Neither blocks nor log segments found. Creating new log.
I20260924 05:00:30.051620 11641 log.cc:824] T 00000000000000000000000000000000 P 8389a786572b4aaabc19b179ed4d20fc: Log is configured to *not* fsync() on all Append() calls
I20260924 05:00:30.052371 11641 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8389a786572b4aaabc19b179ed4d20fc: No bootstrap required, opened a new log
I20260924 05:00:30.055148 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 11573
I20260924 05:00:30.055238 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/minicluster-data/master-0/wal/instance
I20260924 05:00:30.055126 11641 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8389a786572b4aaabc19b179ed4d20fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8389a786572b4aaabc19b179ed4d20fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39797 } }
I20260924 05:00:30.055298 11641 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8389a786572b4aaabc19b179ed4d20fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:00:30.055328 11641 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8389a786572b4aaabc19b179ed4d20fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8389a786572b4aaabc19b179ed4d20fc, State: Initialized, Role: FOLLOWER
I20260924 05:00:30.055402 392 external_mini_cluster.cc:1428] Setting key d4a9c79e292a37976b1f5b236536ec6d
I20260924 05:00:30.055465 11641 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8389a786572b4aaabc19b179ed4d20fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8389a786572b4aaabc19b179ed4d20fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39797 } }
I20260924 05:00:30.055526 11641 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8389a786572b4aaabc19b179ed4d20fc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:00:30.055549 11641 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8389a786572b4aaabc19b179ed4d20fc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:00:30.055584 11641 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8389a786572b4aaabc19b179ed4d20fc [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:00:30.056268 11641 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8389a786572b4aaabc19b179ed4d20fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8389a786572b4aaabc19b179ed4d20fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39797 } }
I20260924 05:00:30.056377 11641 leader_election.cc:302] T 00000000000000000000000000000000 P 8389a786572b4aaabc19b179ed4d20fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8389a786572b4aaabc19b179ed4d20fc; no voters:
I20260924 05:00:30.056593 11641 leader_election.cc:288] T 00000000000000000000000000000000 P 8389a786572b4aaabc19b179ed4d20fc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:00:30.056813 11647 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8389a786572b4aaabc19b179ed4d20fc [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:00:30.056898 11641 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8389a786572b4aaabc19b179ed4d20fc [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:00:30.057776 11647 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8389a786572b4aaabc19b179ed4d20fc [term 1 LEADER]: Becoming Leader. State: Replica: 8389a786572b4aaabc19b179ed4d20fc, State: Running, Role: LEADER
I20260924 05:00:30.057934 11647 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8389a786572b4aaabc19b179ed4d20fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8389a786572b4aaabc19b179ed4d20fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39797 } }
I20260924 05:00:30.058315 11647 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8389a786572b4aaabc19b179ed4d20fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8389a786572b4aaabc19b179ed4d20fc. Latest consensus state: current_term: 1 leader_uuid: "8389a786572b4aaabc19b179ed4d20fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8389a786572b4aaabc19b179ed4d20fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39797 } } }
I20260924 05:00:30.058456 11647 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8389a786572b4aaabc19b179ed4d20fc [sys.catalog]: This master's current role is: LEADER
I20260924 05:00:30.058671 11660 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:00:30.059422 11646 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8389a786572b4aaabc19b179ed4d20fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8389a786572b4aaabc19b179ed4d20fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8389a786572b4aaabc19b179ed4d20fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39797 } } }
I20260924 05:00:30.059494 11646 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8389a786572b4aaabc19b179ed4d20fc [sys.catalog]: This master's current role is: LEADER
I20260924 05:00:30.059859 11641 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 05:00:30.060204 11660 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 24 05:00:30 dist-test-slave-70st krb5kdc[11129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226029, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:00:30.068151 11660 catalog_manager.cc:1347] Generated new cluster ID: 55442cfcbe014c889dd299174e4fd94c
I20260924 05:00:30.068221 11660 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:00:30.068745 11645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:00:30.055885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48912 (local address 127.0.98.62:39797)
0924 05:00:30.056496 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0924 05:00:30.056506 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 05:00:30.056592 (+ 86us) server_negotiation.cc:407] Connection header received
0924 05:00:30.056737 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:00:30.056742 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:00:30.056796 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:00:30.056972 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:00:30.057682 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:00:30.059075 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:00:30.060186 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:00:30.060773 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:00:30.060902 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:00:30.063939 (+ 3037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:00:30.063983 (+ 44us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:00:30.063983 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:00:30.064078 (+ 95us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:00:30.066682 (+ 2604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:00:30.067069 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:00:30.067073 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:00:30.067074 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:00:30.067184 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:00:30.067522 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:00:30.067523 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:00:30.067524 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:00:30.067866 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:00:30.068031 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0924 05:00:30.068156 (+ 125us) server_negotiation.cc:299] Negotiation successful
0924 05:00:30.068523 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":512,"threads_started":1}
I20260924 05:00:30.094138 11660 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:00:30.094964 11660 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:00:30.100011 11660 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8389a786572b4aaabc19b179ed4d20fc: Generated new TSK 0
I20260924 05:00:30.100265 11660 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 05:00:30.783978 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:46783/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39797
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:00:30.868472 11686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:00:30.868674 11686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:00:30.868696 11686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:00:30.869935 11686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:00:30.869982 11686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:00:30.870008 11686 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 05:00:30.871464 11686 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:46783/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39797
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11686
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:00:30.871673 11686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:00:30.871901 11686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:00:30.877842 11691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:00:30.877837 11694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:00:30.877929 11692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:00:30.878656 11686 server_base.cc:1034] running on GCE node
I20260924 05:00:30.878868 11686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:00:30.879109 11686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:00:30.880271 11686 hybrid_clock.cc:648] HybridClock initialized: now 1790226030880250 us; error 33 us; skew 500 ppm
Sep 24 05:00:30 dist-test-slave-70st krb5kdc[11129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226030, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:00:30.882761 11686 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 05:00:30.883162 11686 webserver.cc:460] Webserver started at http://127.0.98.1:44581/ using document root <none> and password file <none>
I20260924 05:00:30.883384 11686 fs_manager.cc:359] Metadata directory not provided
I20260924 05:00:30.883432 11686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:00:30.883538 11686 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 05:00:30 dist-test-slave-70st krb5kdc[11129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226030, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 05:00:30.897130 11686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/minicluster-data/ts-0/data/instance:
uuid: "d98d6fdd68ca44e5886c19500f758529"
format_stamp: "Formatted at 2026-09-24 05:00:30 on dist-test-slave-70st"
server_key: "80faa0c671d3d1d5e83a792591015e48"
server_key_iv: "b8178b18ff07e3ff92ca294696d63056"
server_key_version: "kuduclusterkey@0"
I20260924 05:00:30.897734 11686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d98d6fdd68ca44e5886c19500f758529"
format_stamp: "Formatted at 2026-09-24 05:00:30 on dist-test-slave-70st"
server_key: "80faa0c671d3d1d5e83a792591015e48"
server_key_iv: "b8178b18ff07e3ff92ca294696d63056"
server_key_version: "kuduclusterkey@0"
I20260924 05:00:30.903733 11686 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.000s
I20260924 05:00:30.917085 11701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:00:30.917261 11686 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 05:00:30.917351 11686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/minicluster-data/ts-0/wal
uuid: "d98d6fdd68ca44e5886c19500f758529"
format_stamp: "Formatted at 2026-09-24 05:00:30 on dist-test-slave-70st"
server_key: "80faa0c671d3d1d5e83a792591015e48"
server_key_iv: "b8178b18ff07e3ff92ca294696d63056"
server_key_version: "kuduclusterkey@0"
I20260924 05:00:30.917459 11686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:00:30.941071 11686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:00:30.943444 11686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:00:30.943586 11686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:00:30.943873 11686 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:00:30.944260 11686 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:00:30.944314 11686 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:00:30.944342 11686 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:00:30.944404 11686 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:00:30.951171 11686 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44297
I20260924 05:00:30.951251 11816 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44297 every 8 connection(s)
I20260924 05:00:30.951764 11686 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/minicluster-data/ts-0/data/info.pb
I20260924 05:00:30.954435 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 11686
I20260924 05:00:30.954527 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/minicluster-data/ts-0/wal/instance
I20260924 05:00:30.954687 392 external_mini_cluster.cc:1428] Setting key aad08aec5bf9fbffc210530fbb2b7462
Sep 24 05:00:30 dist-test-slave-70st krb5kdc[11129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226030, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:00:30.961447 11820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:00:30.952580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36403 (local address 127.0.98.62:39797)
0924 05:00:30.953363 (+ 783us) server_negotiation.cc:206] Beginning negotiation
0924 05:00:30.953365 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:00:30.953371 (+ 6us) server_negotiation.cc:407] Connection header received
0924 05:00:30.953391 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:00:30.953392 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:00:30.953401 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:00:30.953482 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:00:30.954112 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:00:30.954816 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:00:30.956234 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:00:30.956537 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:00:30.956589 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:00:30.958854 (+ 2265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:00:30.958856 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:00:30.958856 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:00:30.958863 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:00:30.960173 (+ 1310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:00:30.960539 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:00:30.960540 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:00:30.960541 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:00:30.960596 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:00:30.960810 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:00:30.960811 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:00:30.960812 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:00:30.960919 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:00:30.960975 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 05:00:30.961235 (+ 260us) server_negotiation.cc:299] Negotiation successful
0924 05:00:30.961316 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":758,"thread_start_us":79,"threads_started":1}
I20260924 05:00:30.961602 11819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:00:30.952562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39797 (local address 127.0.98.1:36403)
0924 05:00:30.952727 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 05:00:30.952770 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 05:00:30.953227 (+ 457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:00:30.953615 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:00:30.953620 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:00:30.953815 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:00:30.954019 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:00:30.954023 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:00:30.954919 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:00:30.954920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:00:30.956141 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:00:30.956145 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:00:30.956654 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:00:30.956655 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:00:30.956808 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:00:30.957400 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:00:30.957409 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:00:30.958771 (+ 1362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:00:30.960251 (+ 1480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:00:30.960258 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:00:30.960260 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:00:30.960467 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:00:30.960649 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:00:30.960649 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:00:30.960650 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:00:30.960759 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:00:30.960971 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:00:30.960977 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:00:30.961123 (+ 146us) client_negotiation.cc:769] Sending connection context
0924 05:00:30.961210 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 05:00:30.961353 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":80,"threads_started":1}
I20260924 05:00:30.962129 11817 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39797
I20260924 05:00:30.962231 11817 heartbeater.cc:464] Registering TS with master...
I20260924 05:00:30.962406 11817 heartbeater.cc:511] Master 127.0.98.62:39797 requested a full tablet report, sending...
I20260924 05:00:30.962909 11605 ts_manager.cc:194] Registered new tserver with Master: d98d6fdd68ca44e5886c19500f758529 (127.0.98.1:44297)
I20260924 05:00:30.963500 11605 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:36403
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 05:00:30.992240 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:46783/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39797
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:00:31.096565 11823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:00:31.096796 11823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:00:31.096829 11823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:00:31.097990 11823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:00:31.098031 11823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:00:31.098064 11823 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 05:00:31.099484 11823 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:46783/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39797
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11823
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:00:31.099697 11823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:00:31.099917 11823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:00:31.105506 11830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:00:31.105509 11831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:00:31.105655 11833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:00:31.106501 11823 server_base.cc:1034] running on GCE node
I20260924 05:00:31.106688 11823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:00:31.106941 11823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:00:31.108096 11823 hybrid_clock.cc:648] HybridClock initialized: now 1790226031108076 us; error 34 us; skew 500 ppm
Sep 24 05:00:31 dist-test-slave-70st krb5kdc[11129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226031, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:00:31.110613 11823 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 05:00:31.110996 11823 webserver.cc:460] Webserver started at http://127.0.98.2:32871/ using document root <none> and password file <none>
I20260924 05:00:31.111213 11823 fs_manager.cc:359] Metadata directory not provided
I20260924 05:00:31.111294 11823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:00:31.111418 11823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 05:00:31 dist-test-slave-70st krb5kdc[11129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226031, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 05:00:31.124835 11823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/minicluster-data/ts-1/data/instance:
uuid: "f84064345b71475c8a52f61fa660ed9e"
format_stamp: "Formatted at 2026-09-24 05:00:31 on dist-test-slave-70st"
server_key: "fc6fb7d894dd3b4bd0a5ab4208576a1e"
server_key_iv: "ba232a3cc596a5e049e21568afb424f5"
server_key_version: "kuduclusterkey@0"
I20260924 05:00:31.125308 11823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f84064345b71475c8a52f61fa660ed9e"
format_stamp: "Formatted at 2026-09-24 05:00:31 on dist-test-slave-70st"
server_key: "fc6fb7d894dd3b4bd0a5ab4208576a1e"
server_key_iv: "ba232a3cc596a5e049e21568afb424f5"
server_key_version: "kuduclusterkey@0"
I20260924 05:00:31.127177 11823 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.001s
Sep 24 05:00:31 dist-test-slave-70st krb5kdc[11129](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790226031, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:00:31.148159 11840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:00:31.148432 11823 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.000s sys 0.001s
I20260924 05:00:31.148541 11823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/minicluster-data/ts-1/wal
uuid: "f84064345b71475c8a52f61fa660ed9e"
format_stamp: "Formatted at 2026-09-24 05:00:31 on dist-test-slave-70st"
server_key: "fc6fb7d894dd3b4bd0a5ab4208576a1e"
server_key_iv: "ba232a3cc596a5e049e21568afb424f5"
server_key_version: "kuduclusterkey@0"
I20260924 05:00:31.148655 11823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:00:31.158872 11823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:00:31.162057 11823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:00:31.162285 11823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:00:31.162513 11823 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:00:31.162850 11823 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:00:31.162886 11823 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:00:31.162916 11823 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:00:31.162962 11823 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:00:31.170019 11823 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42295
I20260924 05:00:31.170068 11953 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42295 every 8 connection(s)
I20260924 05:00:31.170483 11823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/minicluster-data/ts-1/data/info.pb
Sep 24 05:00:31 dist-test-slave-70st krb5kdc[11129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226031, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:00:31.177830 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 11823
I20260924 05:00:31.177939 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/minicluster-data/ts-1/wal/instance
I20260924 05:00:31.178128 392 external_mini_cluster.cc:1428] Setting key d6459df2bef71161fa8f8168227d4034
I20260924 05:00:31.180622 11820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:00:31.171232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56327 (local address 127.0.98.62:39797)
0924 05:00:31.171302 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 05:00:31.171304 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:00:31.171461 (+ 157us) server_negotiation.cc:407] Connection header received
0924 05:00:31.171925 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:00:31.171926 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:00:31.171939 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:00:31.172013 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:00:31.172541 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:00:31.173321 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:00:31.174826 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:00:31.175052 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:00:31.175126 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:00:31.177301 (+ 2175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:00:31.177303 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:00:31.177303 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:00:31.177310 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:00:31.178979 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:00:31.179499 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:00:31.179500 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:00:31.179500 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:00:31.179597 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:00:31.179855 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:00:31.179856 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:00:31.179856 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:00:31.179992 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:00:31.180052 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 05:00:31.180439 (+ 387us) server_negotiation.cc:299] Negotiation successful
0924 05:00:31.180531 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 05:00:31.181337 11956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:00:31.171185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39797 (local address 127.0.98.2:56327)
0924 05:00:31.171330 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 05:00:31.171372 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 05:00:31.171845 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:00:31.172091 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:00:31.172095 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:00:31.172279 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:00:31.172470 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:00:31.172475 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:00:31.173443 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:00:31.173444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:00:31.174711 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:00:31.174714 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:00:31.175143 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:00:31.175144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:00:31.175280 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:00:31.175813 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:00:31.175822 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:00:31.177208 (+ 1386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:00:31.179091 (+ 1883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:00:31.179098 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:00:31.179100 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:00:31.179339 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:00:31.179663 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:00:31.179664 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:00:31.179664 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:00:31.179792 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:00:31.180078 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:00:31.180086 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:00:31.180282 (+ 196us) client_negotiation.cc:769] Sending connection context
0924 05:00:31.180979 (+ 697us) client_negotiation.cc:240] Negotiation successful
0924 05:00:31.181138 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":73,"threads_started":1}
I20260924 05:00:31.181789 11954 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39797
I20260924 05:00:31.181895 11954 heartbeater.cc:464] Registering TS with master...
I20260924 05:00:31.182077 11954 heartbeater.cc:511] Master 127.0.98.62:39797 requested a full tablet report, sending...
I20260924 05:00:31.182456 11605 ts_manager.cc:194] Registered new tserver with Master: f84064345b71475c8a52f61fa660ed9e (127.0.98.2:42295)
I20260924 05:00:31.182925 11605 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:56327
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 05:00:31.204524 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:46783/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39797
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:00:31.323304 11960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:00:31.323582 11960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:00:31.323629 11960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:00:31.324810 11960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:00:31.324849 11960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:00:31.324887 11960 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 05:00:31.326642 11960 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:46783/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39797
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11960
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:00:31.326880 11960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:00:31.327098 11960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:00:31.332677 11968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:00:31.333086 11965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:00:31.333335 11966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:00:31.333832 11960 server_base.cc:1034] running on GCE node
I20260924 05:00:31.334095 11960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:00:31.334376 11960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:00:31.337641 11960 hybrid_clock.cc:648] HybridClock initialized: now 1790226031337606 us; error 43 us; skew 500 ppm
Sep 24 05:00:31 dist-test-slave-70st krb5kdc[11129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226031, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:00:31.340850 11960 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 05:00:31.341356 11960 webserver.cc:460] Webserver started at http://127.0.98.3:34279/ using document root <none> and password file <none>
I20260924 05:00:31.341634 11960 fs_manager.cc:359] Metadata directory not provided
I20260924 05:00:31.341686 11960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:00:31.341814 11960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 05:00:31 dist-test-slave-70st krb5kdc[11129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226031, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 05:00:31.353488 11960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/minicluster-data/ts-2/data/instance:
uuid: "cef6ed81f4284dc199c4ca167b3b2b79"
format_stamp: "Formatted at 2026-09-24 05:00:31 on dist-test-slave-70st"
server_key: "4a8c82359f496633339f7e6484953a8a"
server_key_iv: "f41eb01e74b5e36af792cddd66f3ab2c"
server_key_version: "kuduclusterkey@0"
I20260924 05:00:31.353914 11960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/minicluster-data/ts-2/wal/instance:
uuid: "cef6ed81f4284dc199c4ca167b3b2b79"
format_stamp: "Formatted at 2026-09-24 05:00:31 on dist-test-slave-70st"
server_key: "4a8c82359f496633339f7e6484953a8a"
server_key_iv: "f41eb01e74b5e36af792cddd66f3ab2c"
server_key_version: "kuduclusterkey@0"
I20260924 05:00:31.357748 11960 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.002s
I20260924 05:00:31.372206 11975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:00:31.372484 11960 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 05:00:31.372625 11960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/minicluster-data/ts-2/wal
uuid: "cef6ed81f4284dc199c4ca167b3b2b79"
format_stamp: "Formatted at 2026-09-24 05:00:31 on dist-test-slave-70st"
server_key: "4a8c82359f496633339f7e6484953a8a"
server_key_iv: "f41eb01e74b5e36af792cddd66f3ab2c"
server_key_version: "kuduclusterkey@0"
I20260924 05:00:31.372762 11960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:00:31.422430 11960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:00:31.424512 11960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:00:31.424744 11960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:00:31.425001 11960 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:00:31.425350 11960 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:00:31.425391 11960 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:00:31.425446 11960 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:00:31.425488 11960 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:00:31.432538 11960 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34665
I20260924 05:00:31.432991 11960 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/minicluster-data/ts-2/data/info.pb
I20260924 05:00:31.433954 12088 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34665 every 8 connection(s)
Sep 24 05:00:31 dist-test-slave-70st krb5kdc[11129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226031, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:00:31.442992 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 11960
I20260924 05:00:31.443077 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/minicluster-data/ts-2/wal/instance
I20260924 05:00:31.443248 392 external_mini_cluster.cc:1428] Setting key 60a6a81fb5634c1919b5544eaebf10a0
I20260924 05:00:31.445108 11820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:00:31.435448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58941 (local address 127.0.98.62:39797)
0924 05:00:31.435488 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 05:00:31.435490 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:00:31.435500 (+ 10us) server_negotiation.cc:407] Connection header received
0924 05:00:31.436094 (+ 594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:00:31.436096 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:00:31.436107 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:00:31.436155 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:00:31.436709 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:00:31.437479 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:00:31.438704 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:00:31.438910 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:00:31.438964 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:00:31.440930 (+ 1966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:00:31.440932 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:00:31.440932 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:00:31.440939 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:00:31.442213 (+ 1274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:00:31.442568 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:00:31.442569 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:00:31.442569 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:00:31.442621 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:00:31.443457 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:00:31.443458 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:00:31.443459 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:00:31.443593 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:00:31.443636 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 05:00:31.444924 (+ 1288us) server_negotiation.cc:299] Negotiation successful
0924 05:00:31.445015 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 05:00:31.445860 12091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:00:31.434712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39797 (local address 127.0.98.3:58941)
0924 05:00:31.435310 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0924 05:00:31.435363 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 05:00:31.435981 (+ 618us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:00:31.436231 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:00:31.436235 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:00:31.436424 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:00:31.436613 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:00:31.436618 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:00:31.437571 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:00:31.437572 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:00:31.438632 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:00:31.438635 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:00:31.438976 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:00:31.438977 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:00:31.439117 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:00:31.439661 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:00:31.439669 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:00:31.440850 (+ 1181us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:00:31.442293 (+ 1443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:00:31.442299 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:00:31.442301 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:00:31.442505 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:00:31.442672 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:00:31.442672 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:00:31.442673 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:00:31.442787 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:00:31.444607 (+ 1820us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:00:31.444613 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:00:31.444759 (+ 146us) client_negotiation.cc:769] Sending connection context
0924 05:00:31.445450 (+ 691us) client_negotiation.cc:240] Negotiation successful
0924 05:00:31.445619 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":499,"threads_started":1}
I20260924 05:00:31.446476 12089 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39797
I20260924 05:00:31.446609 12089 heartbeater.cc:464] Registering TS with master...
I20260924 05:00:31.446803 12089 heartbeater.cc:511] Master 127.0.98.62:39797 requested a full tablet report, sending...
I20260924 05:00:31.447150 11605 ts_manager.cc:194] Registered new tserver with Master: cef6ed81f4284dc199c4ca167b3b2b79 (127.0.98.3:34665)
I20260924 05:00:31.447511 11605 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:58941
I20260924 05:00:31.455384 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 05:00:31.462623 11820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:00:31.456685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48918 (local address 127.0.98.62:39797)
0924 05:00:31.456729 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 05:00:31.456731 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:00:31.456917 (+ 186us) server_negotiation.cc:407] Connection header received
0924 05:00:31.456971 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:00:31.456972 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:00:31.456983 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:00:31.457027 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:00:31.457659 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:00:31.458176 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:00:31.458938 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:00:31.459133 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:00:31.459183 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:00:31.460288 (+ 1105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:00:31.460289 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:00:31.460290 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:00:31.460296 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:00:31.461590 (+ 1294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:00:31.461879 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:00:31.461880 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:00:31.461880 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:00:31.461933 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:00:31.462114 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:00:31.462115 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:00:31.462115 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:00:31.462235 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:00:31.462296 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 05:00:31.462468 (+ 172us) server_negotiation.cc:299] Negotiation successful
0924 05:00:31.462532 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 05:00:31.464411 11605 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48918:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 05:00:31.466509 11605 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 05:00:31.477861 12104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:00:31.469295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44297 (local address 127.0.0.1:51138)
0924 05:00:31.469660 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0924 05:00:31.469703 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 05:00:31.469767 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:00:31.473709 (+ 3942us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:00:31.473713 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:00:31.473715 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:00:31.473938 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:00:31.473941 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:00:31.475234 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:00:31.475235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:00:31.476484 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:00:31.476486 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:00:31.477490 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:00:31.477491 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:00:31.477586 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:00:31.477656 (+ 70us) client_negotiation.cc:769] Sending connection context
0924 05:00:31.477724 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 05:00:31.477776 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":79,"threads_started":1}
I20260924 05:00:31.478256 12103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:00:31.469388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51138 (local address 127.0.98.1:44297)
0924 05:00:31.473563 (+ 4175us) server_negotiation.cc:206] Beginning negotiation
0924 05:00:31.473567 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 05:00:31.473577 (+ 10us) server_negotiation.cc:407] Connection header received
0924 05:00:31.473606 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:00:31.473608 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:00:31.473621 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:00:31.473696 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:00:31.474023 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:00:31.475145 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:00:31.476567 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:00:31.477302 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:00:31.477365 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:00:31.477458 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 05:00:31.478139 (+ 681us) server_negotiation.cc:299] Negotiation successful
0924 05:00:31.478180 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4143,"thread_start_us":74,"threads_started":1}
I20260924 05:00:31.479147 11751 tablet_service.cc:1467] Processing CreateTablet for tablet 9980b768fac2405d8f6d67d1c528ff27 (DEFAULT_TABLE table=test-table [id=512914cda4fd4494935d2c67c2ea536c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:00:31.479501 11751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9980b768fac2405d8f6d67d1c528ff27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:00:31.481536 12101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:00:31.469548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42295 (local address 127.0.0.1:36486)
0924 05:00:31.473908 (+ 4360us) negotiation.cc:107] Waiting for socket to connect
0924 05:00:31.473917 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:00:31.473958 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:00:31.474553 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:00:31.474554 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:00:31.474555 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:00:31.474603 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:00:31.474604 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:00:31.476094 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:00:31.476095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:00:31.479690 (+ 3595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:00:31.479693 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:00:31.481295 (+ 1602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:00:31.481296 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:00:31.481347 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:00:31.481352 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 05:00:31.481407 (+ 55us) client_negotiation.cc:240] Negotiation successful
0924 05:00:31.481463 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4347,"thread_start_us":59,"threads_started":1}
I20260924 05:00:31.481860 12106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:00:31.469908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36486 (local address 127.0.98.2:42295)
0924 05:00:31.470316 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0924 05:00:31.470319 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:00:31.474014 (+ 3695us) server_negotiation.cc:407] Connection header received
0924 05:00:31.474326 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:00:31.474330 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:00:31.474347 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:00:31.474427 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:00:31.474664 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:00:31.475728 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:00:31.479799 (+ 4071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:00:31.480881 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:00:31.480949 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:00:31.481018 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 05:00:31.481740 (+ 722us) server_negotiation.cc:299] Negotiation successful
0924 05:00:31.481782 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":80,"threads_started":1}
I20260924 05:00:31.483130 12107 tablet_bootstrap.cc:492] T 9980b768fac2405d8f6d67d1c528ff27 P d98d6fdd68ca44e5886c19500f758529: Bootstrap starting.
I20260924 05:00:31.483520 12105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:00:31.469664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53372 (local address 127.0.98.3:34665)
0924 05:00:31.470236 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0924 05:00:31.470241 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 05:00:31.473865 (+ 3624us) server_negotiation.cc:407] Connection header received
0924 05:00:31.475857 (+ 1992us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:00:31.475860 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:00:31.475877 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:00:31.475924 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:00:31.477100 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:00:31.478193 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:00:31.482172 (+ 3979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:00:31.482920 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:00:31.482968 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:00:31.483175 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0924 05:00:31.483394 (+ 219us) server_negotiation.cc:299] Negotiation successful
0924 05:00:31.483439 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":79,"threads_started":1}
I20260924 05:00:31.484373 12107 tablet_bootstrap.cc:654] T 9980b768fac2405d8f6d67d1c528ff27 P d98d6fdd68ca44e5886c19500f758529: Neither blocks nor log segments found. Creating new log.
I20260924 05:00:31.484689 12107 log.cc:824] T 9980b768fac2405d8f6d67d1c528ff27 P d98d6fdd68ca44e5886c19500f758529: Log is configured to *not* fsync() on all Append() calls
I20260924 05:00:31.484855 11882 tablet_service.cc:1467] Processing CreateTablet for tablet 9980b768fac2405d8f6d67d1c528ff27 (DEFAULT_TABLE table=test-table [id=512914cda4fd4494935d2c67c2ea536c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:00:31.485183 11882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9980b768fac2405d8f6d67d1c528ff27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:00:31.487668 12109 tablet_bootstrap.cc:492] T 9980b768fac2405d8f6d67d1c528ff27 P f84064345b71475c8a52f61fa660ed9e: Bootstrap starting.
I20260924 05:00:31.484098 12102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:00:31.469295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34665 (local address 127.0.0.1:53372)
0924 05:00:31.473801 (+ 4506us) negotiation.cc:107] Waiting for socket to connect
0924 05:00:31.473811 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:00:31.474499 (+ 688us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:00:31.476012 (+ 1513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:00:31.476013 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:00:31.476014 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:00:31.476046 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:00:31.476048 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:00:31.478291 (+ 2243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:00:31.478292 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:00:31.481218 (+ 2926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:00:31.481220 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:00:31.483236 (+ 2016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:00:31.483237 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:00:31.483297 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:00:31.483302 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 05:00:31.483975 (+ 673us) client_negotiation.cc:240] Negotiation successful
0924 05:00:31.484022 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4489,"mutex_wait_us":2,"thread_start_us":80,"threads_started":1}
I20260924 05:00:31.488689 12109 tablet_bootstrap.cc:654] T 9980b768fac2405d8f6d67d1c528ff27 P f84064345b71475c8a52f61fa660ed9e: Neither blocks nor log segments found. Creating new log.
I20260924 05:00:31.488947 12023 tablet_service.cc:1467] Processing CreateTablet for tablet 9980b768fac2405d8f6d67d1c528ff27 (DEFAULT_TABLE table=test-table [id=512914cda4fd4494935d2c67c2ea536c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:00:31.489040 12109 log.cc:824] T 9980b768fac2405d8f6d67d1c528ff27 P f84064345b71475c8a52f61fa660ed9e: Log is configured to *not* fsync() on all Append() calls
I20260924 05:00:31.489261 12023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9980b768fac2405d8f6d67d1c528ff27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:00:31.489768 12109 tablet_bootstrap.cc:492] T 9980b768fac2405d8f6d67d1c528ff27 P f84064345b71475c8a52f61fa660ed9e: No bootstrap required, opened a new log
I20260924 05:00:31.489823 12109 ts_tablet_manager.cc:1397] T 9980b768fac2405d8f6d67d1c528ff27 P f84064345b71475c8a52f61fa660ed9e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 05:00:31.491678 12109 raft_consensus.cc:348] T 9980b768fac2405d8f6d67d1c528ff27 P f84064345b71475c8a52f61fa660ed9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cef6ed81f4284dc199c4ca167b3b2b79" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34665 } } peers { permanent_uuid: "d98d6fdd68ca44e5886c19500f758529" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44297 } } peers { permanent_uuid: "f84064345b71475c8a52f61fa660ed9e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42295 } }
I20260924 05:00:31.491847 12109 raft_consensus.cc:374] T 9980b768fac2405d8f6d67d1c528ff27 P f84064345b71475c8a52f61fa660ed9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:00:31.491894 12109 raft_consensus.cc:729] T 9980b768fac2405d8f6d67d1c528ff27 P f84064345b71475c8a52f61fa660ed9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f84064345b71475c8a52f61fa660ed9e, State: Initialized, Role: FOLLOWER
I20260924 05:00:31.492010 12109 consensus_queue.cc:260] T 9980b768fac2405d8f6d67d1c528ff27 P f84064345b71475c8a52f61fa660ed9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cef6ed81f4284dc199c4ca167b3b2b79" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34665 } } peers { permanent_uuid: "d98d6fdd68ca44e5886c19500f758529" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44297 } } peers { permanent_uuid: "f84064345b71475c8a52f61fa660ed9e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42295 } }
I20260924 05:00:31.492257 12109 ts_tablet_manager.cc:1428] T 9980b768fac2405d8f6d67d1c528ff27 P f84064345b71475c8a52f61fa660ed9e: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 05:00:31.492395 11954 heartbeater.cc:503] Master 127.0.98.62:39797 was elected leader, sending a full tablet report...
I20260924 05:00:31.492702 12112 tablet_bootstrap.cc:492] T 9980b768fac2405d8f6d67d1c528ff27 P cef6ed81f4284dc199c4ca167b3b2b79: Bootstrap starting.
I20260924 05:00:31.494107 12112 tablet_bootstrap.cc:654] T 9980b768fac2405d8f6d67d1c528ff27 P cef6ed81f4284dc199c4ca167b3b2b79: Neither blocks nor log segments found. Creating new log.
I20260924 05:00:31.494354 12112 log.cc:824] T 9980b768fac2405d8f6d67d1c528ff27 P cef6ed81f4284dc199c4ca167b3b2b79: Log is configured to *not* fsync() on all Append() calls
I20260924 05:00:31.495005 12112 tablet_bootstrap.cc:492] T 9980b768fac2405d8f6d67d1c528ff27 P cef6ed81f4284dc199c4ca167b3b2b79: No bootstrap required, opened a new log
I20260924 05:00:31.495069 12112 ts_tablet_manager.cc:1397] T 9980b768fac2405d8f6d67d1c528ff27 P cef6ed81f4284dc199c4ca167b3b2b79: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 05:00:31.496732 12112 raft_consensus.cc:348] T 9980b768fac2405d8f6d67d1c528ff27 P cef6ed81f4284dc199c4ca167b3b2b79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cef6ed81f4284dc199c4ca167b3b2b79" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34665 } } peers { permanent_uuid: "d98d6fdd68ca44e5886c19500f758529" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44297 } } peers { permanent_uuid: "f84064345b71475c8a52f61fa660ed9e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42295 } }
I20260924 05:00:31.496858 12112 raft_consensus.cc:374] T 9980b768fac2405d8f6d67d1c528ff27 P cef6ed81f4284dc199c4ca167b3b2b79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:00:31.496889 12112 raft_consensus.cc:729] T 9980b768fac2405d8f6d67d1c528ff27 P cef6ed81f4284dc199c4ca167b3b2b79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cef6ed81f4284dc199c4ca167b3b2b79, State: Initialized, Role: FOLLOWER
I20260924 05:00:31.496992 12112 consensus_queue.cc:260] T 9980b768fac2405d8f6d67d1c528ff27 P cef6ed81f4284dc199c4ca167b3b2b79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cef6ed81f4284dc199c4ca167b3b2b79" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34665 } } peers { permanent_uuid: "d98d6fdd68ca44e5886c19500f758529" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44297 } } peers { permanent_uuid: "f84064345b71475c8a52f61fa660ed9e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42295 } }
I20260924 05:00:31.497239 12112 ts_tablet_manager.cc:1428] T 9980b768fac2405d8f6d67d1c528ff27 P cef6ed81f4284dc199c4ca167b3b2b79: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 05:00:31.497251 12089 heartbeater.cc:503] Master 127.0.98.62:39797 was elected leader, sending a full tablet report...
I20260924 05:00:31.497776 12107 tablet_bootstrap.cc:492] T 9980b768fac2405d8f6d67d1c528ff27 P d98d6fdd68ca44e5886c19500f758529: No bootstrap required, opened a new log
I20260924 05:00:31.497849 12107 ts_tablet_manager.cc:1397] T 9980b768fac2405d8f6d67d1c528ff27 P d98d6fdd68ca44e5886c19500f758529: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.001s
I20260924 05:00:31.499819 12107 raft_consensus.cc:348] T 9980b768fac2405d8f6d67d1c528ff27 P d98d6fdd68ca44e5886c19500f758529 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cef6ed81f4284dc199c4ca167b3b2b79" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34665 } } peers { permanent_uuid: "d98d6fdd68ca44e5886c19500f758529" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44297 } } peers { permanent_uuid: "f84064345b71475c8a52f61fa660ed9e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42295 } }
I20260924 05:00:31.499933 12107 raft_consensus.cc:374] T 9980b768fac2405d8f6d67d1c528ff27 P d98d6fdd68ca44e5886c19500f758529 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:00:31.499950 12107 raft_consensus.cc:729] T 9980b768fac2405d8f6d67d1c528ff27 P d98d6fdd68ca44e5886c19500f758529 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d98d6fdd68ca44e5886c19500f758529, State: Initialized, Role: FOLLOWER
I20260924 05:00:31.500034 12107 consensus_queue.cc:260] T 9980b768fac2405d8f6d67d1c528ff27 P d98d6fdd68ca44e5886c19500f758529 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cef6ed81f4284dc199c4ca167b3b2b79" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34665 } } peers { permanent_uuid: "d98d6fdd68ca44e5886c19500f758529" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44297 } } peers { permanent_uuid: "f84064345b71475c8a52f61fa660ed9e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42295 } }
I20260924 05:00:31.500249 12107 ts_tablet_manager.cc:1428] T 9980b768fac2405d8f6d67d1c528ff27 P d98d6fdd68ca44e5886c19500f758529: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 05:00:31.500293 11817 heartbeater.cc:503] Master 127.0.98.62:39797 was elected leader, sending a full tablet report...
Sep 24 05:00:31 dist-test-slave-70st krb5kdc[11129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226031, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 05:00:31.620944 12111 raft_consensus.cc:482] T 9980b768fac2405d8f6d67d1c528ff27 P f84064345b71475c8a52f61fa660ed9e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 05:00:31.621093 12111 raft_consensus.cc:504] T 9980b768fac2405d8f6d67d1c528ff27 P f84064345b71475c8a52f61fa660ed9e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cef6ed81f4284dc199c4ca167b3b2b79" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34665 } } peers { permanent_uuid: "d98d6fdd68ca44e5886c19500f758529" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44297 } } peers { permanent_uuid: "f84064345b71475c8a52f61fa660ed9e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42295 } }
I20260924 05:00:31.621563 12111 leader_election.cc:288] T 9980b768fac2405d8f6d67d1c528ff27 P f84064345b71475c8a52f61fa660ed9e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cef6ed81f4284dc199c4ca167b3b2b79 (127.0.98.3:34665), d98d6fdd68ca44e5886c19500f758529 (127.0.98.1:44297)
I20260924 05:00:31.622812 12115 raft_consensus.cc:482] T 9980b768fac2405d8f6d67d1c528ff27 P d98d6fdd68ca44e5886c19500f758529 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 05:00:31.622931 12115 raft_consensus.cc:504] T 9980b768fac2405d8f6d67d1c528ff27 P d98d6fdd68ca44e5886c19500f758529 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cef6ed81f4284dc199c4ca167b3b2b79" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34665 } } peers { permanent_uuid: "d98d6fdd68ca44e5886c19500f758529" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44297 } } peers { permanent_uuid: "f84064345b71475c8a52f61fa660ed9e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42295 } }
I20260924 05:00:31.623229 12115 leader_election.cc:288] T 9980b768fac2405d8f6d67d1c528ff27 P d98d6fdd68ca44e5886c19500f758529 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cef6ed81f4284dc199c4ca167b3b2b79 (127.0.98.3:34665), f84064345b71475c8a52f61fa660ed9e (127.0.98.2:42295)
I20260924 05:00:31.626243 11956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:00:31.621851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34665 (local address 127.0.98.2:48075)
0924 05:00:31.622271 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0924 05:00:31.622284 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 05:00:31.622331 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:00:31.622802 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:00:31.622804 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:00:31.622806 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:00:31.622875 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:00:31.622877 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:00:31.624050 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:00:31.624052 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:00:31.625261 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:00:31.625264 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:00:31.626010 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:00:31.626011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:00:31.626086 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:00:31.626089 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:00:31.626131 (+ 42us) client_negotiation.cc:240] Negotiation successful
0924 05:00:31.626169 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399}
I20260924 05:00:31.626883 12105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:00:31.622454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48075 (local address 127.0.98.3:34665)
0924 05:00:31.622497 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 05:00:31.622500 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:00:31.622508 (+ 8us) server_negotiation.cc:407] Connection header received
0924 05:00:31.622534 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:00:31.622535 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:00:31.622547 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:00:31.622596 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:00:31.622960 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:00:31.623641 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:00:31.625358 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:00:31.625907 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:00:31.625971 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:00:31.625981 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 05:00:31.626776 (+ 795us) server_negotiation.cc:299] Negotiation successful
0924 05:00:31.626817 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 05:00:31.627137 12030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9980b768fac2405d8f6d67d1c528ff27" candidate_uuid: "f84064345b71475c8a52f61fa660ed9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cef6ed81f4284dc199c4ca167b3b2b79" is_pre_election: true
I20260924 05:00:31.627280 12030 raft_consensus.cc:2454] T 9980b768fac2405d8f6d67d1c528ff27 P cef6ed81f4284dc199c4ca167b3b2b79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f84064345b71475c8a52f61fa660ed9e in term 0.
I20260924 05:00:31.627681 11843 leader_election.cc:302] T 9980b768fac2405d8f6d67d1c528ff27 P f84064345b71475c8a52f61fa660ed9e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cef6ed81f4284dc199c4ca167b3b2b79, f84064345b71475c8a52f61fa660ed9e; no voters:
I20260924 05:00:31.627859 12111 raft_consensus.cc:2781] T 9980b768fac2405d8f6d67d1c528ff27 P f84064345b71475c8a52f61fa660ed9e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 05:00:31.627912 12111 raft_consensus.cc:482] T 9980b768fac2405d8f6d67d1c528ff27 P f84064345b71475c8a52f61fa660ed9e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 05:00:31.627933 12111 raft_consensus.cc:3037] T 9980b768fac2405d8f6d67d1c528ff27 P f84064345b71475c8a52f61fa660ed9e [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:00:31.628813 12111 raft_consensus.cc:504] T 9980b768fac2405d8f6d67d1c528ff27 P f84064345b71475c8a52f61fa660ed9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cef6ed81f4284dc199c4ca167b3b2b79" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34665 } } peers { permanent_uuid: "d98d6fdd68ca44e5886c19500f758529" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44297 } } peers { permanent_uuid: "f84064345b71475c8a52f61fa660ed9e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42295 } }
I20260924 05:00:31.628937 12111 leader_election.cc:288] T 9980b768fac2405d8f6d67d1c528ff27 P f84064345b71475c8a52f61fa660ed9e [CANDIDATE]: Term 1 election: Requested vote from peers cef6ed81f4284dc199c4ca167b3b2b79 (127.0.98.3:34665), d98d6fdd68ca44e5886c19500f758529 (127.0.98.1:44297)
I20260924 05:00:31.629199 12030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9980b768fac2405d8f6d67d1c528ff27" candidate_uuid: "f84064345b71475c8a52f61fa660ed9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cef6ed81f4284dc199c4ca167b3b2b79"
I20260924 05:00:31.629268 12030 raft_consensus.cc:3037] T 9980b768fac2405d8f6d67d1c528ff27 P cef6ed81f4284dc199c4ca167b3b2b79 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:00:31.630208 12030 raft_consensus.cc:2454] T 9980b768fac2405d8f6d67d1c528ff27 P cef6ed81f4284dc199c4ca167b3b2b79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f84064345b71475c8a52f61fa660ed9e in term 1.
I20260924 05:00:31.630411 11843 leader_election.cc:302] T 9980b768fac2405d8f6d67d1c528ff27 P f84064345b71475c8a52f61fa660ed9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cef6ed81f4284dc199c4ca167b3b2b79, f84064345b71475c8a52f61fa660ed9e; no voters:
I20260924 05:00:31.630514 12111 raft_consensus.cc:2781] T 9980b768fac2405d8f6d67d1c528ff27 P f84064345b71475c8a52f61fa660ed9e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:00:31.630635 12111 raft_consensus.cc:686] T 9980b768fac2405d8f6d67d1c528ff27 P f84064345b71475c8a52f61fa660ed9e [term 1 LEADER]: Becoming Leader. State: Replica: f84064345b71475c8a52f61fa660ed9e, State: Running, Role: LEADER
I20260924 05:00:31.630712 12111 consensus_queue.cc:237] T 9980b768fac2405d8f6d67d1c528ff27 P f84064345b71475c8a52f61fa660ed9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cef6ed81f4284dc199c4ca167b3b2b79" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34665 } } peers { permanent_uuid: "d98d6fdd68ca44e5886c19500f758529" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44297 } } peers { permanent_uuid: "f84064345b71475c8a52f61fa660ed9e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42295 } }
I20260924 05:00:31.631501 11605 catalog_manager.cc:5495] T 9980b768fac2405d8f6d67d1c528ff27 P f84064345b71475c8a52f61fa660ed9e reported cstate change: term changed from 0 to 1, leader changed from <none> to f84064345b71475c8a52f61fa660ed9e (127.0.98.2). New cstate: current_term: 1 leader_uuid: "f84064345b71475c8a52f61fa660ed9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cef6ed81f4284dc199c4ca167b3b2b79" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34665 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d98d6fdd68ca44e5886c19500f758529" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f84064345b71475c8a52f61fa660ed9e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42295 } health_report { overall_health: HEALTHY } } }
I20260924 05:00:31.632936 12105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:00:31.623582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59245 (local address 127.0.98.3:34665)
0924 05:00:31.626951 (+ 3369us) server_negotiation.cc:206] Beginning negotiation
0924 05:00:31.626952 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:00:31.629584 (+ 2632us) server_negotiation.cc:407] Connection header received
0924 05:00:31.629676 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:00:31.629677 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:00:31.629684 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:00:31.629709 (+ 25us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:00:31.629898 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:00:31.630601 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:00:31.632031 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:00:31.632588 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:00:31.632629 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:00:31.632638 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 05:00:31.632837 (+ 199us) server_negotiation.cc:299] Negotiation successful
0924 05:00:31.632870 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3356,"thread_start_us":64,"threads_started":1}
I20260924 05:00:31.633298 12119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:00:31.623423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34665 (local address 127.0.98.1:59245)
0924 05:00:31.629510 (+ 6087us) negotiation.cc:107] Waiting for socket to connect
0924 05:00:31.629523 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 05:00:31.629611 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:00:31.629761 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:00:31.629762 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:00:31.629763 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:00:31.629835 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:00:31.629837 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:00:31.630689 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:00:31.630690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:00:31.631936 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:00:31.631939 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:00:31.632692 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:00:31.632693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:00:31.632764 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:00:31.632768 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 05:00:31.633157 (+ 389us) client_negotiation.cc:240] Negotiation successful
0924 05:00:31.633201 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6067,"thread_start_us":83,"threads_started":1}
I20260924 05:00:31.634034 12106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:00:31.623990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54889 (local address 127.0.98.2:42295)
0924 05:00:31.624023 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 05:00:31.624024 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:00:31.629688 (+ 5664us) server_negotiation.cc:407] Connection header received
0924 05:00:31.629771 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:00:31.629772 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:00:31.629781 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:00:31.629804 (+ 23us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:00:31.630037 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:00:31.630635 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:00:31.633151 (+ 2516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:00:31.633692 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:00:31.633734 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:00:31.633743 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 05:00:31.633946 (+ 203us) server_negotiation.cc:299] Negotiation successful
0924 05:00:31.633977 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 05:00:31.635142 11956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:00:31.621964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44297 (local address 127.0.98.2:51951)
0924 05:00:31.626309 (+ 4345us) negotiation.cc:107] Waiting for socket to connect
0924 05:00:31.626319 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:00:31.626357 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:00:31.631957 (+ 5600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:00:31.631958 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:00:31.631960 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:00:31.631992 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:00:31.631993 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:00:31.633238 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:00:31.633239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:00:31.634199 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:00:31.634201 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:00:31.634928 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:00:31.634929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:00:31.635004 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:00:31.635007 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:00:31.635044 (+ 37us) client_negotiation.cc:240] Negotiation successful
0924 05:00:31.635076 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4335,"thread_start_us":112,"threads_started":1}
I20260924 05:00:31.634351 12117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:00:31.623828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42295 (local address 127.0.98.1:54889)
0924 05:00:31.629621 (+ 5793us) negotiation.cc:107] Waiting for socket to connect
0924 05:00:31.629632 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 05:00:31.629711 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:00:31.629871 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:00:31.629871 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:00:31.629873 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:00:31.629963 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:00:31.629965 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:00:31.630740 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:00:31.630741 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:00:31.633062 (+ 2321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:00:31.633065 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:00:31.633794 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:00:31.633795 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:00:31.633868 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:00:31.633871 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:00:31.634210 (+ 339us) client_negotiation.cc:240] Negotiation successful
0924 05:00:31.634252 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5772,"mutex_wait_us":1243,"thread_start_us":84,"threads_started":1}
I20260924 05:00:31.636355 12030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9980b768fac2405d8f6d67d1c528ff27" candidate_uuid: "d98d6fdd68ca44e5886c19500f758529" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cef6ed81f4284dc199c4ca167b3b2b79" is_pre_election: true
I20260924 05:00:31.635460 12103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:00:31.622682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51951 (local address 127.0.98.1:44297)
0924 05:00:31.622728 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 05:00:31.622730 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:00:31.626489 (+ 3759us) server_negotiation.cc:407] Connection header received
0924 05:00:31.626514 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:00:31.626515 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:00:31.626525 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:00:31.626567 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:00:31.632072 (+ 5505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:00:31.632779 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:00:31.634306 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:00:31.634825 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:00:31.634888 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:00:31.634897 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 05:00:31.635343 (+ 446us) server_negotiation.cc:299] Negotiation successful
0924 05:00:31.635396 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":23}
I20260924 05:00:31.636456 12030 raft_consensus.cc:2379] T 9980b768fac2405d8f6d67d1c528ff27 P cef6ed81f4284dc199c4ca167b3b2b79 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d98d6fdd68ca44e5886c19500f758529 in current term 1: Already voted for candidate f84064345b71475c8a52f61fa660ed9e in this term.
I20260924 05:00:31.636658 11771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9980b768fac2405d8f6d67d1c528ff27" candidate_uuid: "f84064345b71475c8a52f61fa660ed9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d98d6fdd68ca44e5886c19500f758529" is_pre_election: true
I20260924 05:00:31.636698 11908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9980b768fac2405d8f6d67d1c528ff27" candidate_uuid: "d98d6fdd68ca44e5886c19500f758529" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f84064345b71475c8a52f61fa660ed9e" is_pre_election: true
I20260924 05:00:31.636767 11771 raft_consensus.cc:2454] T 9980b768fac2405d8f6d67d1c528ff27 P d98d6fdd68ca44e5886c19500f758529 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f84064345b71475c8a52f61fa660ed9e in term 0.
I20260924 05:00:31.636876 11770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9980b768fac2405d8f6d67d1c528ff27" candidate_uuid: "f84064345b71475c8a52f61fa660ed9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d98d6fdd68ca44e5886c19500f758529"
I20260924 05:00:31.636948 11770 raft_consensus.cc:3037] T 9980b768fac2405d8f6d67d1c528ff27 P d98d6fdd68ca44e5886c19500f758529 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:00:31.637549 11702 leader_election.cc:302] T 9980b768fac2405d8f6d67d1c528ff27 P d98d6fdd68ca44e5886c19500f758529 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d98d6fdd68ca44e5886c19500f758529; no voters: cef6ed81f4284dc199c4ca167b3b2b79, f84064345b71475c8a52f61fa660ed9e
I20260924 05:00:31.638101 11770 raft_consensus.cc:2454] T 9980b768fac2405d8f6d67d1c528ff27 P d98d6fdd68ca44e5886c19500f758529 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f84064345b71475c8a52f61fa660ed9e in term 1.
I20260924 05:00:31.638180 12115 raft_consensus.cc:2726] T 9980b768fac2405d8f6d67d1c528ff27 P d98d6fdd68ca44e5886c19500f758529 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 05:00:31.656940 12106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:00:31.653868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36492 (local address 127.0.98.2:42295)
0924 05:00:31.653911 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 05:00:31.653913 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:00:31.653923 (+ 10us) server_negotiation.cc:407] Connection header received
0924 05:00:31.654004 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:00:31.654005 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:00:31.654030 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:00:31.654106 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:00:31.654244 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:00:31.655000 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:00:31.656010 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:00:31.656327 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:00:31.656388 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:00:31.656559 (+ 171us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:00:31.656692 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:00:31.656787 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 05:00:31.656805 (+ 18us) server_negotiation.cc:299] Negotiation successful
0924 05:00:31.656827 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 05:00:31.659348 11770 raft_consensus.cc:1261] T 9980b768fac2405d8f6d67d1c528ff27 P d98d6fdd68ca44e5886c19500f758529 [term 1 FOLLOWER]: Refusing update from remote peer f84064345b71475c8a52f61fa660ed9e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:00:31.659384 12030 raft_consensus.cc:1261] T 9980b768fac2405d8f6d67d1c528ff27 P cef6ed81f4284dc199c4ca167b3b2b79 [term 1 FOLLOWER]: Refusing update from remote peer f84064345b71475c8a52f61fa660ed9e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:00:31.659740 12120 consensus_queue.cc:1035] T 9980b768fac2405d8f6d67d1c528ff27 P f84064345b71475c8a52f61fa660ed9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d98d6fdd68ca44e5886c19500f758529" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:00:31.659917 12111 consensus_queue.cc:1035] T 9980b768fac2405d8f6d67d1c528ff27 P f84064345b71475c8a52f61fa660ed9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cef6ed81f4284dc199c4ca167b3b2b79" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:00:31.663558 12125 mvcc.cc:205] Tried to move back new op lower bound from 7332765825673498624 to 7332765825559990272. Current Snapshot: MvccSnapshot[applied={T|T < 7332765825673498624}]
I20260924 05:00:31.665045 12124 mvcc.cc:205] Tried to move back new op lower bound from 7332765825673498624 to 7332765825559990272. Current Snapshot: MvccSnapshot[applied={T|T < 7332765825673498624}]
I20260924 05:00:31.665252 12123 mvcc.cc:205] Tried to move back new op lower bound from 7332765825673498624 to 7332765825559990272. Current Snapshot: MvccSnapshot[applied={T|T < 7332765825673498624}]
I20260924 05:00:31.671491 11604 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48918:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:00:31.671613 11604 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48918:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:00:31.672408 11604 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8389a786572b4aaabc19b179ed4d20fc: Sending DeleteTablet for 3 replicas of tablet 9980b768fac2405d8f6d67d1c528ff27
I20260924 05:00:31.672983 12023 tablet_service.cc:1514] Processing DeleteTablet for tablet 9980b768fac2405d8f6d67d1c528ff27 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:00:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53372
I20260924 05:00:31.673293 11751 tablet_service.cc:1514] Processing DeleteTablet for tablet 9980b768fac2405d8f6d67d1c528ff27 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:00:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51138
I20260924 05:00:31.673522 11882 tablet_service.cc:1514] Processing DeleteTablet for tablet 9980b768fac2405d8f6d67d1c528ff27 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:00:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36486
I20260924 05:00:31.674459 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 11686
I20260924 05:00:31.676096 12135 tablet_replica.cc:330] T 9980b768fac2405d8f6d67d1c528ff27 P f84064345b71475c8a52f61fa660ed9e: stopping tablet replica
I20260924 05:00:31.676234 12135 raft_consensus.cc:2229] T 9980b768fac2405d8f6d67d1c528ff27 P f84064345b71475c8a52f61fa660ed9e [term 1 LEADER]: Raft consensus shutting down.
I20260924 05:00:31.676429 12135 raft_consensus.cc:2258] T 9980b768fac2405d8f6d67d1c528ff27 P f84064345b71475c8a52f61fa660ed9e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:00:31.676859 12133 tablet_replica.cc:330] T 9980b768fac2405d8f6d67d1c528ff27 P cef6ed81f4284dc199c4ca167b3b2b79: stopping tablet replica
I20260924 05:00:31.676998 12135 ts_tablet_manager.cc:1905] T 9980b768fac2405d8f6d67d1c528ff27 P f84064345b71475c8a52f61fa660ed9e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:00:31.677068 12133 raft_consensus.cc:2229] T 9980b768fac2405d8f6d67d1c528ff27 P cef6ed81f4284dc199c4ca167b3b2b79 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:00:31.677254 12133 raft_consensus.cc:2258] T 9980b768fac2405d8f6d67d1c528ff27 P cef6ed81f4284dc199c4ca167b3b2b79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:00:31.678526 12133 ts_tablet_manager.cc:1905] T 9980b768fac2405d8f6d67d1c528ff27 P cef6ed81f4284dc199c4ca167b3b2b79: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:00:31.679339 12135 ts_tablet_manager.cc:1918] T 9980b768fac2405d8f6d67d1c528ff27 P f84064345b71475c8a52f61fa660ed9e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 05:00:31.679464 12135 log.cc:1196] T 9980b768fac2405d8f6d67d1c528ff27 P f84064345b71475c8a52f61fa660ed9e: Deleting WAL directory at /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/minicluster-data/ts-1/wal/wals/9980b768fac2405d8f6d67d1c528ff27
I20260924 05:00:31.679839 12135 ts_tablet_manager.cc:1939] T 9980b768fac2405d8f6d67d1c528ff27 P f84064345b71475c8a52f61fa660ed9e: Deleting consensus metadata
I20260924 05:00:31.680331 11589 catalog_manager.cc:4841] TS f84064345b71475c8a52f61fa660ed9e (127.0.98.2:42295): tablet 9980b768fac2405d8f6d67d1c528ff27 (table test-table [id=512914cda4fd4494935d2c67c2ea536c]) successfully deleted
I20260924 05:00:31.681229 12133 ts_tablet_manager.cc:1918] T 9980b768fac2405d8f6d67d1c528ff27 P cef6ed81f4284dc199c4ca167b3b2b79: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 05:00:31.681354 12133 log.cc:1196] T 9980b768fac2405d8f6d67d1c528ff27 P cef6ed81f4284dc199c4ca167b3b2b79: Deleting WAL directory at /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790225929848825-392-0/minicluster-data/ts-2/wal/wals/9980b768fac2405d8f6d67d1c528ff27
I20260924 05:00:31.681991 12133 ts_tablet_manager.cc:1939] T 9980b768fac2405d8f6d67d1c528ff27 P cef6ed81f4284dc199c4ca167b3b2b79: Deleting consensus metadata
I20260924 05:00:31.682436 11591 catalog_manager.cc:4841] TS cef6ed81f4284dc199c4ca167b3b2b79 (127.0.98.3:34665): tablet 9980b768fac2405d8f6d67d1c528ff27 (table test-table [id=512914cda4fd4494935d2c67c2ea536c]) successfully deleted
W20260924 05:00:31.686522 11589 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44297 (error 108)
I20260924 05:00:31.687358 12102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:00:31.687236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44297 (local address 127.0.0.1:51152)
0924 05:00:31.687287 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0924 05:00:31.687323 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260924 05:00:31.687582 11589 catalog_manager.cc:4584] TS d98d6fdd68ca44e5886c19500f758529 (127.0.98.1:44297): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9980b768fac2405d8f6d67d1c528ff27: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44297: connect: Connection refused (error 111)
I20260924 05:00:31.687830 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 11823
I20260924 05:00:31.693879 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 11960
I20260924 05:00:31.697686 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 11573
I20260924 05:00:31.702482 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 05:00:32.036355 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 05:00:32.036471 392 mini_ranger.cc:66] Stopping Ranger...
I20260924 05:00:32.380405 392 mini_ranger.cc:68] Stopped Ranger
2026-09-24 05:00:32.380 UTC [11159] LOG: received smart shutdown request
2026-09-24 05:00:32.383 UTC [11159] LOG: background worker "logical replication launcher" (PID 11168) exited with exit code 1
2026-09-24 05:00:32.384 UTC [11163] LOG: shutting down
2026-09-24 05:00:32.548 UTC [11159] LOG: database system is shut down
2026-09-24T05:00:32Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (45575 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:00:32 dist-test-slave-70st krb5kdc[12145](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:00:32 dist-test-slave-70st krb5kdc[12145](info): set up 2 sockets
Sep 24 05:00:32 dist-test-slave-70st krb5kdc[12145](info): commencing operation
krb5kdc: starting...
W20260924 05:00:33.581216 392 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/krb5kdc/test-user.keytab.
Sep 24 05:00:33 dist-test-slave-70st krb5kdc[12145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226033, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:00:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:00:33Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260924 05:00:33.738162 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskZbcc3B/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/postgres -l logfile start
2026-09-24 05:00:36.115 UTC [12175] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 05:00:36.115 UTC [12175] LOG: listening on IPv4 address "127.0.98.20", port 45805
2026-09-24 05:00:36.118 UTC [12175] LOG: listening on Unix socket "/tmp/.s.PGSQL.45805"
2026-09-24 05:00:36.145 UTC [12178] LOG: database system was shut down at 2026-09-24 05:00:34 UTC
2026-09-24 05:00:36.148 UTC [12175] LOG: database system is ready to accept connections
I20260924 05:00:37.013015 392 mini_postgres.cc:96] Postgres bound to 45805
2026-09-24 05:00:37.018 UTC [12186] FATAL: database "slave" does not exist
127.0.98.20:45805 - accepting connections
I20260924 05:00:37.019307 392 mini_ranger.cc:161] Starting Ranger...
I20260924 05:00:37.033998 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 05:00:37.461917 392 mini_ranger.cc:87] Created ranger Postgres database
I20260924 05:00:37.461999 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ranger-admin
2026-09-24 05:00:37,727 [I] DB FLAVOR :POSTGRES
2026-09-24 05:00:37,727 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 05:00:37,727 [I] Checking connection..
2026-09-24 05:00:37,727 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZbcc3B/build/release/bin/postgresql.jar:/tmp/dist-test-taskZbcc3B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45805/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 05:00:37,960 [I] Checking connection passed.
2026-09-24 05:00:37,961 [I] --------- Verifying version history table ---------
2026-09-24 05:00:37,961 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZbcc3B/build/release/bin/postgresql.jar:/tmp/dist-test-taskZbcc3B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45805/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 05:00:38,200 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 05:00:38,200 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZbcc3B/build/release/bin/postgresql.jar:/tmp/dist-test-taskZbcc3B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45805/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 05:00:38,430 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 05:00:38,430 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 05:00:38,430 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZbcc3B/build/release/bin/postgresql.jar:/tmp/dist-test-taskZbcc3B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45805/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskZbcc3B/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 05:00:38.679 UTC [12250] WARNING: there is no transaction in progress
2026-09-24 05:00:38,684 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 05:00:38,685 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZbcc3B/build/release/bin/postgresql.jar:/tmp/dist-test-taskZbcc3B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45805/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 05:00:38,917 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 05:00:38,917 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 05:00:38,917 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZbcc3B/build/release/bin/postgresql.jar:/tmp/dist-test-taskZbcc3B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45805/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 05:00:39,156 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZbcc3B/build/release/bin/postgresql.jar:/tmp/dist-test-taskZbcc3B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45805/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 05:00:39,391 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZbcc3B/build/release/bin/postgresql.jar:/tmp/dist-test-taskZbcc3B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45805/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-70st.c.gcp-upstream.internal','N') ;"
2026-09-24 05:00:39,630 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZbcc3B/build/release/bin/postgresql.jar:/tmp/dist-test-taskZbcc3B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45805/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 05:00:39,870 [I] Table x_portal_user does not exist in database ranger
2026-09-24 05:00:39,870 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZbcc3B/build/release/bin/postgresql.jar:/tmp/dist-test-taskZbcc3B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45805/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 05:00:40,120 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 05:00:40,120 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZbcc3B/build/release/bin/postgresql.jar:/tmp/dist-test-taskZbcc3B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45805/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 05:00:40,365 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZbcc3B/build/release/bin/postgresql.jar:/tmp/dist-test-taskZbcc3B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45805/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 05:00:40,596 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 05:00:40,596 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZbcc3B/build/release/bin/postgresql.jar:/tmp/dist-test-taskZbcc3B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45805/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskZbcc3B/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 05:00:41.490 UTC [12358] WARNING: there is no transaction in progress
2026-09-24 05:00:41.506 UTC [12358] WARNING: there is no transaction in progress
2026-09-24 05:00:41.522 UTC [12358] WARNING: there is no transaction in progress
2026-09-24 05:00:41.537 UTC [12358] WARNING: there is no transaction in progress
2026-09-24 05:00:41.552 UTC [12358] WARNING: there is no transaction in progress
2026-09-24 05:00:41.568 UTC [12358] WARNING: there is no transaction in progress
2026-09-24 05:00:41.622 UTC [12358] WARNING: there is no transaction in progress
2026-09-24 05:00:41.629 UTC [12358] WARNING: there is no transaction in progress
2026-09-24 05:00:41.642 UTC [12358] WARNING: there is no transaction in progress
2026-09-24 05:00:41.652 UTC [12358] WARNING: there is no transaction in progress
2026-09-24 05:00:41.662 UTC [12358] WARNING: there is no transaction in progress
2026-09-24 05:00:41.675 UTC [12358] WARNING: there is no transaction in progress
2026-09-24 05:00:41.684 UTC [12358] WARNING: there is no transaction in progress
2026-09-24 05:00:41.690 UTC [12358] WARNING: there is no transaction in progress
2026-09-24 05:00:41.696 UTC [12358] WARNING: there is no transaction in progress
2026-09-24 05:00:41.703 UTC [12358] WARNING: there is no transaction in progress
2026-09-24 05:00:42,326 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 05:00:42,327 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZbcc3B/build/release/bin/postgresql.jar:/tmp/dist-test-taskZbcc3B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45805/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 05:00:42,558 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 05:00:42,559 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZbcc3B/build/release/bin/postgresql.jar:/tmp/dist-test-taskZbcc3B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45805/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 05:00:42,814 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 05:00:42,815 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZbcc3B/build/release/bin/postgresql.jar:/tmp/dist-test-taskZbcc3B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45805/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 05:00:43,060 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 05:00:43,060 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZbcc3B/build/release/bin/postgresql.jar:/tmp/dist-test-taskZbcc3B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45805/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 05:00:43,294 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZbcc3B/build/release/bin/postgresql.jar:/tmp/dist-test-taskZbcc3B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45805/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 05:00:43,527 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZbcc3B/build/release/bin/postgresql.jar:/tmp/dist-test-taskZbcc3B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45805/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-70st.c.gcp-upstream.internal';"
2026-09-24 05:00:43,753 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 05:00:43,753 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZbcc3B/build/release/bin/postgresql.jar:/tmp/dist-test-taskZbcc3B/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45805/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 05:00:43,997 [I] DB_PATCHES have already been applied
I20260924 05:00:44.002826 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskZbcc3B/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskZbcc3B/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskZbcc3B/build/release/bin/postgresql.jar:/tmp/dist-test-taskZbcc3B/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 05:00:44.002893 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260924 05:00:44.005743 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:56857
Sep 24, 2026 5:00:44 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 5:00:44 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 5:00:44 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 5:00:44 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 5:00:44 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 5:00:44 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 5:00:44 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 05:00:44.991393 392 mini_ranger.cc:160] Time spent starting Ranger: real 7.972s user 0.000s sys 0.007s
Sep 24, 2026 5:01:04 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 5:01:04 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 24, 2026 5:01:04 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 5:01:04 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 5:01:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 5:01:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 5:01:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 5:01:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 5:01:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 5:01:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 5:01:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 5:01:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 5:01:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 5:01:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 5:01:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 5:01:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 5:01:04 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 5:01:05 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 05:01:06.345052 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 05:01:06 dist-test-slave-70st krb5kdc[12145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226066, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 05:01:06.423377 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 05:01:06.431578 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 05:01:07.280328 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 05:01:07.280416 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ranger-kms
I20260924 05:01:07.280432 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:45805
2026-09-24 05:01:07,465 [I] DB FLAVOR :POSTGRES
2026-09-24 05:01:07,466 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 05:01:07,466 [I] Checking connection
2026-09-24 05:01:07,466 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZbcc3B/build/release/bin/postgresql.jar:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45805/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 05:01:07,722 [I] connection success
2026-09-24 05:01:07,722 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 05:01:07,722 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 05:01:07,722 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZbcc3B/build/release/bin/postgresql.jar:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45805/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 05:01:07,995 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 05:01:07,996 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 05:01:07,996 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 05:01:07,996 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskZbcc3B/build/release/bin/postgresql.jar:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45805/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 05:01:08,274 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 05:01:08.696762 392 mini_ranger_kms.cc:310] Created kms service
I20260924 05:01:08.895393 392 mini_ranger_kms.cc:326] Created kudu user
I20260924 05:01:08.936355 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 05:01:08 dist-test-slave-70st krb5kdc[12145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226066, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 05:01:09.319290 392 mini_ranger_kms.cc:384] Added ranger policy
I20260924 05:01:09.319406 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskZbcc3B/build/release/bin/postgresql.jar:/tmp/dist-test-taskZbcc3B/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskZbcc3B/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 05:01:09.319422 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260924 05:01:09.322988 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12538
I20260924 05:01:09.323060 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:41535
Sep 24, 2026 5:01:10 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 5:01:10 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 5:01:10 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 5:01:10 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 5:01:10 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 5:01:10 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260924 05:01:11.375613 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.952s user 0.002s sys 0.165s
I20260924 05:01:11.375703 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 05:01:11.375748 392 mini_ranger_kms.cc:401] 127.0.98.20:41535/kms/v1/keys
Sep 24 05:01:11 dist-test-slave-70st krb5kdc[12145](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790226071, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 24 05:01:11 dist-test-slave-70st krb5kdc[12145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226071, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 24 05:01:12 dist-test-slave-70st krb5kdc[12145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226071, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 24 05:01:14 dist-test-slave-70st krb5kdc[12145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226074, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 05:01:14.608675 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33935
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33935
--ranger_config_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:41535/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:01:14.721596 12589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:01:14.721889 12589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:01:14.721920 12589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:01:14.723595 12589 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:01:14.723666 12589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:01:14.723685 12589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:01:14.723701 12589 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:01:14.723717 12589 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:01:14.725987 12589 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:41535/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33935
--ranger_config_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12589
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:01:14.726239 12589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:01:14.726478 12589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:01:14.731550 12594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:01:14.731663 12589 server_base.cc:1034] running on GCE node
W20260924 05:01:14.731599 12595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:01:14.731822 12597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:01:14.732069 12589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:01:14.732379 12589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:01:14.733505 12589 hybrid_clock.cc:648] HybridClock initialized: now 1790226074733487 us; error 31 us; skew 500 ppm
Sep 24 05:01:14 dist-test-slave-70st krb5kdc[12145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226074, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:01:14.744488 12589 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 05:01:14.745020 12589 webserver.cc:460] Webserver started at http://127.0.98.62:44525/ using document root <none> and password file <none>
I20260924 05:01:14.745253 12589 fs_manager.cc:359] Metadata directory not provided
I20260924 05:01:14.745304 12589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:01:14.745460 12589 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 05:01:14 dist-test-slave-70st krb5kdc[12145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226074, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 05:01:14.797758 12589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/master-0/data/instance:
uuid: "4ebc24cba59046c5be83914718165fbc"
format_stamp: "Formatted at 2026-09-24 05:01:14 on dist-test-slave-70st"
server_key: "c4c2cf99c0f5e1d732973c72bf450e89"
server_key_iv: "094f8eee5034fc7a5aa19cb663b74a9e"
server_key_version: "kuduclusterkey@0"
I20260924 05:01:14.798220 12589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/master-0/wal/instance:
uuid: "4ebc24cba59046c5be83914718165fbc"
format_stamp: "Formatted at 2026-09-24 05:01:14 on dist-test-slave-70st"
server_key: "c4c2cf99c0f5e1d732973c72bf450e89"
server_key_iv: "094f8eee5034fc7a5aa19cb663b74a9e"
server_key_version: "kuduclusterkey@0"
I20260924 05:01:14.800022 12589 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260924 05:01:14.813982 12604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:01:14.814208 12589 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 05:01:14.814303 12589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/master-0/wal
uuid: "4ebc24cba59046c5be83914718165fbc"
format_stamp: "Formatted at 2026-09-24 05:01:14 on dist-test-slave-70st"
server_key: "c4c2cf99c0f5e1d732973c72bf450e89"
server_key_iv: "094f8eee5034fc7a5aa19cb663b74a9e"
server_key_version: "kuduclusterkey@0"
I20260924 05:01:14.814402 12589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:01:14.835202 12589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:01:14.837167 12589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:01:14.837314 12589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:01:14.841152 12589 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33935
I20260924 05:01:14.841200 12656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33935 every 8 connection(s)
I20260924 05:01:14.841564 12589 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:01:14.842293 12657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:01:14.845324 12657 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ebc24cba59046c5be83914718165fbc: Bootstrap starting.
I20260924 05:01:14.845332 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 12589
I20260924 05:01:14.845456 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/master-0/wal/instance
I20260924 05:01:14.845620 392 external_mini_cluster.cc:1428] Setting key eee8e5b3eadfcbfd18bd1658956f24a3
I20260924 05:01:14.846040 12657 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ebc24cba59046c5be83914718165fbc: Neither blocks nor log segments found. Creating new log.
I20260924 05:01:14.846413 12657 log.cc:824] T 00000000000000000000000000000000 P 4ebc24cba59046c5be83914718165fbc: Log is configured to *not* fsync() on all Append() calls
I20260924 05:01:14.847173 12657 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ebc24cba59046c5be83914718165fbc: No bootstrap required, opened a new log
I20260924 05:01:14.849606 12657 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ebc24cba59046c5be83914718165fbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ebc24cba59046c5be83914718165fbc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33935 } }
I20260924 05:01:14.849856 12657 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4ebc24cba59046c5be83914718165fbc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:01:14.849907 12657 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ebc24cba59046c5be83914718165fbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ebc24cba59046c5be83914718165fbc, State: Initialized, Role: FOLLOWER
I20260924 05:01:14.850021 12657 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ebc24cba59046c5be83914718165fbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ebc24cba59046c5be83914718165fbc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33935 } }
I20260924 05:01:14.850088 12657 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4ebc24cba59046c5be83914718165fbc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:01:14.850137 12657 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4ebc24cba59046c5be83914718165fbc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:01:14.850199 12657 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ebc24cba59046c5be83914718165fbc [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:01:14.850795 12657 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4ebc24cba59046c5be83914718165fbc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ebc24cba59046c5be83914718165fbc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33935 } }
I20260924 05:01:14.850905 12657 leader_election.cc:302] T 00000000000000000000000000000000 P 4ebc24cba59046c5be83914718165fbc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4ebc24cba59046c5be83914718165fbc; no voters:
I20260924 05:01:14.851087 12657 leader_election.cc:288] T 00000000000000000000000000000000 P 4ebc24cba59046c5be83914718165fbc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:01:14.851138 12662 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4ebc24cba59046c5be83914718165fbc [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:01:14.851321 12662 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4ebc24cba59046c5be83914718165fbc [term 1 LEADER]: Becoming Leader. State: Replica: 4ebc24cba59046c5be83914718165fbc, State: Running, Role: LEADER
I20260924 05:01:14.851397 12657 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ebc24cba59046c5be83914718165fbc [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:01:14.851459 12662 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ebc24cba59046c5be83914718165fbc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ebc24cba59046c5be83914718165fbc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33935 } }
Sep 24 05:01:14 dist-test-slave-70st krb5kdc[12145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226074, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:01:14.851836 12662 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ebc24cba59046c5be83914718165fbc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4ebc24cba59046c5be83914718165fbc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ebc24cba59046c5be83914718165fbc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33935 } } }
I20260924 05:01:14.851971 12662 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ebc24cba59046c5be83914718165fbc [sys.catalog]: This master's current role is: LEADER
I20260924 05:01:14.852138 12662 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ebc24cba59046c5be83914718165fbc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ebc24cba59046c5be83914718165fbc. Latest consensus state: current_term: 1 leader_uuid: "4ebc24cba59046c5be83914718165fbc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ebc24cba59046c5be83914718165fbc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33935 } } }
I20260924 05:01:14.852242 12662 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ebc24cba59046c5be83914718165fbc [sys.catalog]: This master's current role is: LEADER
I20260924 05:01:14.852382 12671 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:01:14.852873 12671 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:01:14.853144 12657 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 05:01:14.854708 12679 mvcc.cc:205] Tried to move back new op lower bound from 7332766002601209856 to 7332766002592141312. Current Snapshot: MvccSnapshot[applied={T|T < 7332766002601209856}]
I20260924 05:01:14.857498 12660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:14.846092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44358 (local address 127.0.98.62:33935)
0924 05:01:14.846261 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 05:01:14.846272 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 05:01:14.846302 (+ 30us) server_negotiation.cc:407] Connection header received
0924 05:01:14.846764 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:01:14.846774 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:01:14.846857 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:01:14.846993 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:01:14.847645 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:14.848491 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:14.849184 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:14.849653 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:14.849748 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:14.852170 (+ 2422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:01:14.852174 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:01:14.852174 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:01:14.852185 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:01:14.854067 (+ 1882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:01:14.854445 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:01:14.854448 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:01:14.854448 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:01:14.854541 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:01:14.854759 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:01:14.854760 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:01:14.854761 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:01:14.856566 (+ 1805us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:01:14.856890 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0924 05:01:14.856974 (+ 84us) server_negotiation.cc:299] Negotiation successful
0924 05:01:14.857252 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":59,"threads_started":1}
I20260924 05:01:14.858539 12671 catalog_manager.cc:1347] Generated new cluster ID: c7075f7034d8426ab06ecbcaa882890c
I20260924 05:01:14.858587 12671 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:01:14.880395 12671 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:01:14.881225 12671 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:01:14.891767 12671 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4ebc24cba59046c5be83914718165fbc: Generated new TSK 0
I20260924 05:01:14.892078 12671 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 05:01:15.521135 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:41535/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33935
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:01:15.612586 12702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:01:15.612785 12702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:01:15.612824 12702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:01:15.614019 12702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:01:15.614058 12702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:01:15.614090 12702 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 05:01:15.615510 12702 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:41535/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12702
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:01:15.615742 12702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:01:15.615957 12702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:01:15.621941 12708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:01:15.622038 12702 server_base.cc:1034] running on GCE node
W20260924 05:01:15.622133 12710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:01:15.621963 12707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:01:15.622406 12702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:01:15.622637 12702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:01:15.623804 12702 hybrid_clock.cc:648] HybridClock initialized: now 1790226075623792 us; error 25 us; skew 500 ppm
Sep 24 05:01:15 dist-test-slave-70st krb5kdc[12145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226075, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:01:15.626282 12702 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 05:01:15.626696 12702 webserver.cc:460] Webserver started at http://127.0.98.1:36941/ using document root <none> and password file <none>
I20260924 05:01:15.626901 12702 fs_manager.cc:359] Metadata directory not provided
I20260924 05:01:15.626943 12702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:01:15.627058 12702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 05:01:15 dist-test-slave-70st krb5kdc[12145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226075, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 05:01:15.641835 12702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ts-0/data/instance:
uuid: "b69c8bdc782a4bd98b0de4c18c92b16d"
format_stamp: "Formatted at 2026-09-24 05:01:15 on dist-test-slave-70st"
server_key: "e1f98c7e1d6aa05e1368b2e691fd975b"
server_key_iv: "a8fa7d87c33bc28f8650227140804fd6"
server_key_version: "kuduclusterkey@0"
I20260924 05:01:15.642417 12702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b69c8bdc782a4bd98b0de4c18c92b16d"
format_stamp: "Formatted at 2026-09-24 05:01:15 on dist-test-slave-70st"
server_key: "e1f98c7e1d6aa05e1368b2e691fd975b"
server_key_iv: "a8fa7d87c33bc28f8650227140804fd6"
server_key_version: "kuduclusterkey@0"
I20260924 05:01:15.644444 12702 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260924 05:01:15.660528 12717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:01:15.660740 12702 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 05:01:15.660840 12702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ts-0/wal
uuid: "b69c8bdc782a4bd98b0de4c18c92b16d"
format_stamp: "Formatted at 2026-09-24 05:01:15 on dist-test-slave-70st"
server_key: "e1f98c7e1d6aa05e1368b2e691fd975b"
server_key_iv: "a8fa7d87c33bc28f8650227140804fd6"
server_key_version: "kuduclusterkey@0"
I20260924 05:01:15.660941 12702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:01:15.684496 12702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:01:15.686806 12702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:01:15.686947 12702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:01:15.687194 12702 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:01:15.687566 12702 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:01:15.687613 12702 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:01:15.687642 12702 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:01:15.687696 12702 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:01:15.694538 12702 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44797
I20260924 05:01:15.694936 12702 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ts-0/data/info.pb
I20260924 05:01:15.696870 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 12702
I20260924 05:01:15.696957 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ts-0/wal/instance
I20260924 05:01:15.697146 392 external_mini_cluster.cc:1428] Setting key cbd3a65437408a74394298ccbbd7bd71
I20260924 05:01:15.704691 12830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44797 every 8 connection(s)
Sep 24 05:01:15 dist-test-slave-70st krb5kdc[12145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226075, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:01:15.720634 12835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:15.705196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33935 (local address 127.0.98.1:44709)
0924 05:01:15.705382 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:15.705457 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0924 05:01:15.706092 (+ 635us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:01:15.711109 (+ 5017us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:01:15.711116 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:01:15.711381 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:01:15.711659 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:01:15.711665 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:01:15.712697 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:01:15.712698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:01:15.713997 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:01:15.714001 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:01:15.714558 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:01:15.714559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:01:15.714705 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:15.715363 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:01:15.715374 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:01:15.717043 (+ 1669us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:01:15.718972 (+ 1929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:01:15.718980 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:01:15.718983 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:01:15.719278 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:01:15.719528 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:01:15.719529 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:01:15.719529 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:01:15.719659 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:01:15.719934 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:01:15.719939 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:01:15.720114 (+ 175us) client_negotiation.cc:769] Sending connection context
0924 05:01:15.720220 (+ 106us) client_negotiation.cc:240] Negotiation successful
0924 05:01:15.720385 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":84,"threads_started":1}
I20260924 05:01:15.721195 12836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:15.705492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44709 (local address 127.0.98.62:33935)
0924 05:01:15.710849 (+ 5357us) server_negotiation.cc:206] Beginning negotiation
0924 05:01:15.710852 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:01:15.710863 (+ 11us) server_negotiation.cc:407] Connection header received
0924 05:01:15.710892 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:01:15.710893 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:01:15.710907 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:01:15.710989 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:01:15.711780 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:15.712586 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:15.714099 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:15.714409 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:15.714475 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:15.717148 (+ 2673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:01:15.717149 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:01:15.717150 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:01:15.717158 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:01:15.718883 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:01:15.719398 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:01:15.719399 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:01:15.719399 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:01:15.719464 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:01:15.719722 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:01:15.719723 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:01:15.719723 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:01:15.719856 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:01:15.719910 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 05:01:15.721007 (+ 1097us) server_negotiation.cc:299] Negotiation successful
0924 05:01:15.721090 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5323,"thread_start_us":75,"threads_started":1}
I20260924 05:01:15.721889 12831 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33935
I20260924 05:01:15.721982 12831 heartbeater.cc:464] Registering TS with master...
I20260924 05:01:15.722162 12831 heartbeater.cc:511] Master 127.0.98.62:33935 requested a full tablet report, sending...
I20260924 05:01:15.722661 12621 ts_manager.cc:194] Registered new tserver with Master: b69c8bdc782a4bd98b0de4c18c92b16d (127.0.98.1:44797)
I20260924 05:01:15.723244 12621 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:44709
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 05:01:15.744520 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:41535/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33935
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:01:15.858773 12839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:01:15.859228 12839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:01:15.859332 12839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:01:15.861294 12839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:01:15.861459 12839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:01:15.861557 12839 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 05:01:15.863853 12839 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:41535/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12839
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:01:15.864302 12839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:01:15.864619 12839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:01:15.871117 12847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:01:15.871096 12846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:01:15.871419 12849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:01:15.872457 12839 server_base.cc:1034] running on GCE node
I20260924 05:01:15.872671 12839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:01:15.872932 12839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:01:15.881491 12839 hybrid_clock.cc:648] HybridClock initialized: now 1790226075881451 us; error 57 us; skew 500 ppm
Sep 24 05:01:15 dist-test-slave-70st krb5kdc[12145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226075, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:01:15.884714 12839 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 05:01:15.885262 12839 webserver.cc:460] Webserver started at http://127.0.98.2:39477/ using document root <none> and password file <none>
I20260924 05:01:15.885568 12839 fs_manager.cc:359] Metadata directory not provided
I20260924 05:01:15.885679 12839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:01:15.885855 12839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 05:01:15 dist-test-slave-70st krb5kdc[12145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226075, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 05:01:15.904052 12839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ts-1/data/instance:
uuid: "cc8a9a07a7c640df97e5d5bbefb75036"
format_stamp: "Formatted at 2026-09-24 05:01:15 on dist-test-slave-70st"
server_key: "e16085b7438518ab1e522909a4e386c9"
server_key_iv: "423ba0ad52f9d3f11866b674b9f9d7bc"
server_key_version: "kuduclusterkey@0"
I20260924 05:01:15.904436 12839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ts-1/wal/instance:
uuid: "cc8a9a07a7c640df97e5d5bbefb75036"
format_stamp: "Formatted at 2026-09-24 05:01:15 on dist-test-slave-70st"
server_key: "e16085b7438518ab1e522909a4e386c9"
server_key_iv: "423ba0ad52f9d3f11866b674b9f9d7bc"
server_key_version: "kuduclusterkey@0"
I20260924 05:01:15.906397 12839 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
Sep 24 05:01:15 dist-test-slave-70st krb5kdc[12145](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790226075, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:01:15.933585 12856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:01:15.937630 12839 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.000s sys 0.001s
I20260924 05:01:15.937774 12839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ts-1/wal
uuid: "cc8a9a07a7c640df97e5d5bbefb75036"
format_stamp: "Formatted at 2026-09-24 05:01:15 on dist-test-slave-70st"
server_key: "e16085b7438518ab1e522909a4e386c9"
server_key_iv: "423ba0ad52f9d3f11866b674b9f9d7bc"
server_key_version: "kuduclusterkey@0"
I20260924 05:01:15.937884 12839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:01:15.961597 12839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:01:15.964179 12839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:01:15.964385 12839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:01:15.964673 12839 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:01:15.965176 12839 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:01:15.965265 12839 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:01:15.965340 12839 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:01:15.965384 12839 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:01:15.972103 12839 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45099
I20260924 05:01:15.972616 12839 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ts-1/data/info.pb
I20260924 05:01:15.975522 12969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45099 every 8 connection(s)
I20260924 05:01:15.981463 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 12839
I20260924 05:01:15.981582 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ts-1/wal/instance
I20260924 05:01:15.981751 392 external_mini_cluster.cc:1428] Setting key cb4aaf9d69af3281347803238ec9ace3
Sep 24 05:01:15 dist-test-slave-70st krb5kdc[12145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226075, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:01:15.987015 12836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:15.976479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59543 (local address 127.0.98.62:33935)
0924 05:01:15.976873 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0924 05:01:15.976875 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:01:15.976890 (+ 15us) server_negotiation.cc:407] Connection header received
0924 05:01:15.977449 (+ 559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:01:15.977451 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:01:15.977464 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:01:15.977495 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:01:15.978379 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:15.979142 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:15.980631 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:15.980911 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:15.981001 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:15.983705 (+ 2704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:01:15.983707 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:01:15.983707 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:01:15.983715 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:01:15.985354 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:01:15.985849 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:01:15.985851 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:01:15.985851 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:01:15.985908 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:01:15.986210 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:01:15.986211 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:01:15.986212 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:01:15.986347 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:01:15.986396 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 05:01:15.986812 (+ 416us) server_negotiation.cc:299] Negotiation successful
0924 05:01:15.986901 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365}
I20260924 05:01:15.987831 12972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:15.976175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33935 (local address 127.0.98.2:59543)
0924 05:01:15.976569 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:15.976621 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 05:01:15.977274 (+ 653us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:01:15.977726 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:01:15.977730 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:01:15.977993 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:01:15.978278 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:01:15.978284 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:01:15.979256 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:01:15.979257 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:01:15.980522 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:01:15.980525 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:01:15.981052 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:01:15.981054 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:01:15.981199 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:15.981853 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:01:15.981863 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:01:15.983597 (+ 1734us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:01:15.985469 (+ 1872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:01:15.985476 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:01:15.985478 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:01:15.985763 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:01:15.985988 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:01:15.985989 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:01:15.985989 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:01:15.986135 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:01:15.986442 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:01:15.986449 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:01:15.986653 (+ 204us) client_negotiation.cc:769] Sending connection context
0924 05:01:15.987365 (+ 712us) client_negotiation.cc:240] Negotiation successful
0924 05:01:15.987536 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":86,"threads_started":1}
I20260924 05:01:15.988499 12970 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33935
I20260924 05:01:15.988596 12970 heartbeater.cc:464] Registering TS with master...
I20260924 05:01:15.988783 12970 heartbeater.cc:511] Master 127.0.98.62:33935 requested a full tablet report, sending...
I20260924 05:01:15.989235 12621 ts_manager.cc:194] Registered new tserver with Master: cc8a9a07a7c640df97e5d5bbefb75036 (127.0.98.2:45099)
I20260924 05:01:15.989768 12621 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:59543
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 05:01:16.019731 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:41535/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33935
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:01:16.138633 12975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:01:16.138908 12975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:01:16.138942 12975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:01:16.140558 12975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:01:16.140610 12975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:01:16.140645 12975 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 05:01:16.142838 12975 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:41535/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12975
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:01:16.143059 12975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:01:16.143339 12975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:01:16.149452 12982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:01:16.149852 12981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:01:16.150084 12984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:01:16.151080 12975 server_base.cc:1034] running on GCE node
I20260924 05:01:16.151368 12975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:01:16.151602 12975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:01:16.152743 12975 hybrid_clock.cc:648] HybridClock initialized: now 1790226076152725 us; error 31 us; skew 500 ppm
Sep 24 05:01:16 dist-test-slave-70st krb5kdc[12145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226076, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:01:16.156075 12975 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 05:01:16.156548 12975 webserver.cc:460] Webserver started at http://127.0.98.3:45683/ using document root <none> and password file <none>
I20260924 05:01:16.156783 12975 fs_manager.cc:359] Metadata directory not provided
I20260924 05:01:16.156829 12975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:01:16.156944 12975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 05:01:16 dist-test-slave-70st krb5kdc[12145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226076, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 05:01:16.173641 12975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ts-2/data/instance:
uuid: "9738e1af1b794ed6915de35b19fb335c"
format_stamp: "Formatted at 2026-09-24 05:01:16 on dist-test-slave-70st"
server_key: "62fa9103d83cbb2c9f38866043aa2256"
server_key_iv: "eac20e147bdb4f58daf059627868d84c"
server_key_version: "kuduclusterkey@0"
I20260924 05:01:16.174177 12975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9738e1af1b794ed6915de35b19fb335c"
format_stamp: "Formatted at 2026-09-24 05:01:16 on dist-test-slave-70st"
server_key: "62fa9103d83cbb2c9f38866043aa2256"
server_key_iv: "eac20e147bdb4f58daf059627868d84c"
server_key_version: "kuduclusterkey@0"
I20260924 05:01:16.186429 12975 fs_manager.cc:691] Time spent creating directory manager: real 0.012s user 0.001s sys 0.000s
I20260924 05:01:16.202732 12991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:01:16.203109 12975 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.000s sys 0.001s
I20260924 05:01:16.203253 12975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ts-2/wal
uuid: "9738e1af1b794ed6915de35b19fb335c"
format_stamp: "Formatted at 2026-09-24 05:01:16 on dist-test-slave-70st"
server_key: "62fa9103d83cbb2c9f38866043aa2256"
server_key_iv: "eac20e147bdb4f58daf059627868d84c"
server_key_version: "kuduclusterkey@0"
I20260924 05:01:16.203418 12975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:01:16.218771 12975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:01:16.227592 12975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:01:16.227723 12975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:01:16.227912 12975 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:01:16.228343 12975 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:01:16.228381 12975 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:01:16.228411 12975 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:01:16.228458 12975 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:01:16.234977 12975 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44295
I20260924 05:01:16.235354 12975 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ts-2/data/info.pb
I20260924 05:01:16.239138 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 12975
I20260924 05:01:16.239233 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ts-2/wal/instance
I20260924 05:01:16.239419 392 external_mini_cluster.cc:1428] Setting key 48d0bb29f2169106b512ac4a6980087c
Sep 24 05:01:16 dist-test-slave-70st krb5kdc[12145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226076, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:01:16.244392 12836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:16.236109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37515 (local address 127.0.98.62:33935)
0924 05:01:16.236157 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 05:01:16.236159 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:01:16.236447 (+ 288us) server_negotiation.cc:407] Connection header received
0924 05:01:16.236943 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:01:16.236944 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:01:16.236956 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:01:16.237009 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:01:16.237602 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:16.238102 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:16.239149 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:16.239442 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:16.239488 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:16.241744 (+ 2256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:01:16.241745 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:01:16.241745 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:01:16.241753 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:01:16.243103 (+ 1350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:01:16.243464 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:01:16.243465 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:01:16.243465 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:01:16.243522 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:01:16.243734 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:01:16.243735 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:01:16.243735 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:01:16.243847 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:01:16.243901 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 05:01:16.244206 (+ 305us) server_negotiation.cc:299] Negotiation successful
0924 05:01:16.244301 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 05:01:16.244565 13107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:16.236129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33935 (local address 127.0.98.3:37515)
0924 05:01:16.236329 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:16.236366 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 05:01:16.236851 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:01:16.237080 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:01:16.237084 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:01:16.237284 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:01:16.237517 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:01:16.237523 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:01:16.238186 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:01:16.238187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:01:16.239041 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:01:16.239044 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:01:16.239597 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:01:16.239597 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:01:16.239813 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:16.240368 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:01:16.240379 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:01:16.241662 (+ 1283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:01:16.243176 (+ 1514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:01:16.243183 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:01:16.243185 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:01:16.243402 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:01:16.243570 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:01:16.243571 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:01:16.243571 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:01:16.243682 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:01:16.243898 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:01:16.243904 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:01:16.244064 (+ 160us) client_negotiation.cc:769] Sending connection context
0924 05:01:16.244185 (+ 121us) client_negotiation.cc:240] Negotiation successful
0924 05:01:16.244330 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":83,"threads_started":1}
I20260924 05:01:16.245087 13105 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33935
I20260924 05:01:16.245184 13105 heartbeater.cc:464] Registering TS with master...
I20260924 05:01:16.245370 13105 heartbeater.cc:511] Master 127.0.98.62:33935 requested a full tablet report, sending...
I20260924 05:01:16.245623 13104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44295 every 8 connection(s)
I20260924 05:01:16.245769 12621 ts_manager.cc:194] Registered new tserver with Master: 9738e1af1b794ed6915de35b19fb335c (127.0.98.3:44295)
I20260924 05:01:16.246143 12621 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:37515
I20260924 05:01:16.262133 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 05:01:16.262920 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 12702
I20260924 05:01:16.270210 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 12839
I20260924 05:01:16.276206 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 12975
I20260924 05:01:16.280265 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 12589
I20260924 05:01:16.285969 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33935
--webserver_interface=127.0.98.62
--webserver_port=44525
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33935
--ranger_config_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:41535/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:41535/kms with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:01:16.366070 13108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:01:16.366268 13108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:01:16.366289 13108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:01:16.367368 13108 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:01:16.367405 13108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:01:16.367417 13108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:01:16.367439 13108 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:01:16.367461 13108 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:01:16.368940 13108 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:41535/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33935
--ranger_config_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=44525
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13108
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:01:16.369153 13108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:01:16.369377 13108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:01:16.373685 13116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:01:16.373817 13114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:01:16.373801 13108 server_base.cc:1034] running on GCE node
W20260924 05:01:16.373682 13113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:01:16.374104 13108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:01:16.374341 13108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:01:16.375488 13108 hybrid_clock.cc:648] HybridClock initialized: now 1790226076375475 us; error 29 us; skew 500 ppm
Sep 24 05:01:16 dist-test-slave-70st krb5kdc[12145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226076, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:01:16.377756 13108 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 05:01:16.378161 13108 webserver.cc:460] Webserver started at http://127.0.98.62:44525/ using document root <none> and password file <none>
I20260924 05:01:16.378381 13108 fs_manager.cc:359] Metadata directory not provided
I20260924 05:01:16.378441 13108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 05:01:16 dist-test-slave-70st krb5kdc[12145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226076, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 05:01:16.434924 13108 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 05:01:16.435623 13124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:01:16.435819 13108 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 05:01:16.435951 13108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/master-0/wal
uuid: "4ebc24cba59046c5be83914718165fbc"
format_stamp: "Formatted at 2026-09-24 05:01:14 on dist-test-slave-70st"
server_key: "c4c2cf99c0f5e1d732973c72bf450e89"
server_key_iv: "094f8eee5034fc7a5aa19cb663b74a9e"
server_key_version: "kuduclusterkey@0"
I20260924 05:01:16.436441 13108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:01:16.458966 13108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:01:16.461203 13108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:01:16.461350 13108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:01:16.466209 13108 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33935
I20260924 05:01:16.466254 13176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33935 every 8 connection(s)
I20260924 05:01:16.466681 13108 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:01:16.467935 13177 sys_catalog.cc:263] Verifying existing consensus state
I20260924 05:01:16.468564 13177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ebc24cba59046c5be83914718165fbc: Bootstrap starting.
I20260924 05:01:16.471884 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 13108
I20260924 05:01:16.472167 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:41535/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:44797
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:33935
--webserver_port=36941
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:41535/kms with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 05:01:16.476543 13177 log.cc:824] T 00000000000000000000000000000000 P 4ebc24cba59046c5be83914718165fbc: Log is configured to *not* fsync() on all Append() calls
I20260924 05:01:16.478189 13177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ebc24cba59046c5be83914718165fbc: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 05:01:16.478425 13177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ebc24cba59046c5be83914718165fbc: Bootstrap complete.
I20260924 05:01:16.480497 13177 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ebc24cba59046c5be83914718165fbc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ebc24cba59046c5be83914718165fbc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33935 } }
I20260924 05:01:16.480715 13177 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ebc24cba59046c5be83914718165fbc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ebc24cba59046c5be83914718165fbc, State: Initialized, Role: FOLLOWER
I20260924 05:01:16.480860 13177 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ebc24cba59046c5be83914718165fbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ebc24cba59046c5be83914718165fbc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33935 } }
I20260924 05:01:16.480921 13177 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4ebc24cba59046c5be83914718165fbc [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:01:16.480960 13177 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4ebc24cba59046c5be83914718165fbc [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:01:16.481001 13177 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ebc24cba59046c5be83914718165fbc [term 1 FOLLOWER]: Advancing to term 2
I20260924 05:01:16.481989 13177 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4ebc24cba59046c5be83914718165fbc [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ebc24cba59046c5be83914718165fbc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33935 } }
I20260924 05:01:16.482076 13177 leader_election.cc:302] T 00000000000000000000000000000000 P 4ebc24cba59046c5be83914718165fbc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4ebc24cba59046c5be83914718165fbc; no voters:
I20260924 05:01:16.482213 13177 leader_election.cc:288] T 00000000000000000000000000000000 P 4ebc24cba59046c5be83914718165fbc [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 05:01:16.482267 13181 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4ebc24cba59046c5be83914718165fbc [term 2 FOLLOWER]: Leader election won for term 2
I20260924 05:01:16.482376 13181 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4ebc24cba59046c5be83914718165fbc [term 2 LEADER]: Becoming Leader. State: Replica: 4ebc24cba59046c5be83914718165fbc, State: Running, Role: LEADER
I20260924 05:01:16.482457 13177 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ebc24cba59046c5be83914718165fbc [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:01:16.482453 13181 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ebc24cba59046c5be83914718165fbc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ebc24cba59046c5be83914718165fbc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33935 } }
I20260924 05:01:16.482854 13181 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ebc24cba59046c5be83914718165fbc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ebc24cba59046c5be83914718165fbc. Latest consensus state: current_term: 2 leader_uuid: "4ebc24cba59046c5be83914718165fbc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ebc24cba59046c5be83914718165fbc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33935 } } }
I20260924 05:01:16.482923 13181 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ebc24cba59046c5be83914718165fbc [sys.catalog]: This master's current role is: LEADER
I20260924 05:01:16.483047 13183 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ebc24cba59046c5be83914718165fbc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "4ebc24cba59046c5be83914718165fbc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ebc24cba59046c5be83914718165fbc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33935 } } }
I20260924 05:01:16.483103 13183 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ebc24cba59046c5be83914718165fbc [sys.catalog]: This master's current role is: LEADER
I20260924 05:01:16.483325 13186 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:01:16.483803 13186 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:01:16.484076 13177 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 05:01:16.484377 13186 catalog_manager.cc:1259] Loaded cluster ID: c7075f7034d8426ab06ecbcaa882890c
I20260924 05:01:16.484429 13186 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:01:16.488130 13186 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:01:16.488613 13186 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4ebc24cba59046c5be83914718165fbc: Loaded TSK: 0
I20260924 05:01:16.488897 13186 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 05:01:16.581086 13179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:01:16.581362 13179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:01:16.581404 13179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:01:16.583276 13179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:01:16.583340 13179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:01:16.583384 13179 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 05:01:16.585662 13179 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:41535/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:44797
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=36941
--tserver_master_addrs=127.0.98.62:33935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13179
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:01:16.585947 13179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:01:16.586217 13179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:01:16.592964 13215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:01:16.594002 13214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:01:16.594208 13179 server_base.cc:1034] running on GCE node
W20260924 05:01:16.594451 13217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:01:16.594766 13179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:01:16.595070 13179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:01:16.596246 13179 hybrid_clock.cc:648] HybridClock initialized: now 1790226076596209 us; error 48 us; skew 500 ppm
Sep 24 05:01:16 dist-test-slave-70st krb5kdc[12145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226076, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:01:16.599522 13179 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 05:01:16.600080 13179 webserver.cc:460] Webserver started at http://127.0.98.1:36941/ using document root <none> and password file <none>
I20260924 05:01:16.600355 13179 fs_manager.cc:359] Metadata directory not provided
I20260924 05:01:16.600438 13179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 05:01:16 dist-test-slave-70st krb5kdc[12145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226076, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 05:01:16.618376 13179 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.000s
I20260924 05:01:16.619666 13225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:01:16.619882 13179 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 05:01:16.619993 13179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ts-0/wal
uuid: "b69c8bdc782a4bd98b0de4c18c92b16d"
format_stamp: "Formatted at 2026-09-24 05:01:15 on dist-test-slave-70st"
server_key: "e1f98c7e1d6aa05e1368b2e691fd975b"
server_key_iv: "a8fa7d87c33bc28f8650227140804fd6"
server_key_version: "kuduclusterkey@0"
I20260924 05:01:16.620450 13179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:01:16.645362 13179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:01:16.647668 13179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:01:16.647817 13179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:01:16.648056 13179 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:01:16.648494 13179 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:01:16.648586 13179 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:01:16.648635 13179 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:01:16.648679 13179 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:01:16.654898 13179 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44797
I20260924 05:01:16.655491 13179 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ts-0/data/info.pb
I20260924 05:01:16.658454 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 13179
I20260924 05:01:16.658720 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:41535/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:45099
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:33935
--webserver_port=39477
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:41535/kms with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 05:01:16.670089 13338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44797 every 8 connection(s)
Sep 24 05:01:16 dist-test-slave-70st krb5kdc[12145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226076, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:01:16.685963 13343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:16.670723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36807 (local address 127.0.98.62:33935)
0924 05:01:16.670942 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0924 05:01:16.670953 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 05:01:16.673744 (+ 2791us) server_negotiation.cc:407] Connection header received
0924 05:01:16.674643 (+ 899us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:01:16.674652 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:01:16.674739 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:01:16.674887 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:01:16.675831 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:16.677272 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:16.678817 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:16.679488 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:16.679615 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:16.682010 (+ 2395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:01:16.682014 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:01:16.682015 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:01:16.682027 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:01:16.684061 (+ 2034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:01:16.684552 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:01:16.684555 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:01:16.684556 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:01:16.684661 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:01:16.684944 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:01:16.684945 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:01:16.684946 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:01:16.685212 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:01:16.685274 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 05:01:16.685626 (+ 352us) server_negotiation.cc:299] Negotiation successful
0924 05:01:16.685754 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":91,"threads_started":1}
I20260924 05:01:16.686815 13342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:16.670324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33935 (local address 127.0.98.1:36807)
0924 05:01:16.673531 (+ 3207us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:16.673595 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0924 05:01:16.674352 (+ 757us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:01:16.675122 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:01:16.675126 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:01:16.675410 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:01:16.675694 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:01:16.675701 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:01:16.677399 (+ 1698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:01:16.677400 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:01:16.678712 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:01:16.678714 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:01:16.679601 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:01:16.679602 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:01:16.679780 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:16.680349 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:01:16.680360 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:01:16.681879 (+ 1519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:01:16.684152 (+ 2273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:01:16.684158 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:01:16.684160 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:01:16.684453 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:01:16.684736 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:01:16.684738 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:01:16.684738 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:01:16.684865 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:01:16.685303 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:01:16.685308 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:01:16.685445 (+ 137us) client_negotiation.cc:769] Sending connection context
0924 05:01:16.686441 (+ 996us) client_negotiation.cc:240] Negotiation successful
0924 05:01:16.686600 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3145,"thread_start_us":117,"threads_started":1}
I20260924 05:01:16.687794 13339 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33935
I20260924 05:01:16.687884 13339 heartbeater.cc:464] Registering TS with master...
I20260924 05:01:16.688056 13339 heartbeater.cc:511] Master 127.0.98.62:33935 requested a full tablet report, sending...
W20260924 05:01:16.688898 13339 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33935 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 05:01:16.688963 13339 heartbeater.cc:464] Registering TS with master...
I20260924 05:01:16.689018 13339 heartbeater.cc:511] Master 127.0.98.62:33935 requested a full tablet report, sending...
I20260924 05:01:16.689345 13339 heartbeater.cc:464] Registering TS with master...
I20260924 05:01:16.689481 13339 heartbeater.cc:511] Master 127.0.98.62:33935 requested a full tablet report, sending...
W20260924 05:01:16.689785 13339 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 05:01:16.761662 13341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:01:16.762004 13341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:01:16.762040 13341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:01:16.763670 13341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:01:16.763732 13341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:01:16.763772 13341 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 05:01:16.765703 13341 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:41535/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:45099
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=39477
--tserver_master_addrs=127.0.98.62:33935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13341
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:01:16.765902 13341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:01:16.766119 13341 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:01:16.771926 13349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:01:16.772034 13348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:01:16.771929 13351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:01:16.773154 13341 server_base.cc:1034] running on GCE node
I20260924 05:01:16.773355 13341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:01:16.773620 13341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:01:16.774799 13341 hybrid_clock.cc:648] HybridClock initialized: now 1790226076774788 us; error 26 us; skew 500 ppm
Sep 24 05:01:16 dist-test-slave-70st krb5kdc[12145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226076, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:01:16.777166 13341 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 05:01:16.777704 13341 webserver.cc:460] Webserver started at http://127.0.98.2:39477/ using document root <none> and password file <none>
I20260924 05:01:16.778014 13341 fs_manager.cc:359] Metadata directory not provided
I20260924 05:01:16.778143 13341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 05:01:16 dist-test-slave-70st krb5kdc[12145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226076, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 05:01:16.799619 13341 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260924 05:01:16.800323 13359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:01:16.800519 13341 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 05:01:16.800611 13341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ts-1/wal
uuid: "cc8a9a07a7c640df97e5d5bbefb75036"
format_stamp: "Formatted at 2026-09-24 05:01:15 on dist-test-slave-70st"
server_key: "e16085b7438518ab1e522909a4e386c9"
server_key_iv: "423ba0ad52f9d3f11866b674b9f9d7bc"
server_key_version: "kuduclusterkey@0"
I20260924 05:01:16.800956 13341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:01:16.816251 13341 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:01:16.818176 13341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:01:16.818323 13341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:01:16.818537 13341 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:01:16.818864 13341 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:01:16.818903 13341 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:01:16.818933 13341 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:01:16.818982 13341 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:01:16.824335 13341 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45099
I20260924 05:01:16.824432 13472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45099 every 8 connection(s)
I20260924 05:01:16.824745 13341 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ts-1/data/info.pb
I20260924 05:01:16.827140 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 13341
I20260924 05:01:16.827380 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:41535/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:44295
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:33935
--webserver_port=45683
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:41535/kms with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 05:01:16 dist-test-slave-70st krb5kdc[12145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226076, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:01:16.839118 13343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:16.826008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53483 (local address 127.0.98.62:33935)
0924 05:01:16.826073 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 05:01:16.826075 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:01:16.829682 (+ 3607us) server_negotiation.cc:407] Connection header received
0924 05:01:16.830249 (+ 567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:01:16.830251 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:01:16.830269 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:01:16.830320 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:01:16.830917 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:16.831563 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:16.832948 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:16.833251 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:16.833298 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:16.835836 (+ 2538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:01:16.835838 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:01:16.835838 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:01:16.835846 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:01:16.837496 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:01:16.837997 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:01:16.837998 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:01:16.837999 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:01:16.838060 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:01:16.838334 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:01:16.838335 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:01:16.838335 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:01:16.838484 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:01:16.838557 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 05:01:16.838926 (+ 369us) server_negotiation.cc:299] Negotiation successful
0924 05:01:16.839013 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 05:01:16.839956 13475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:16.825974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33935 (local address 127.0.98.2:53483)
0924 05:01:16.829522 (+ 3548us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:16.829578 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 05:01:16.830143 (+ 565us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:01:16.830400 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:01:16.830404 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:01:16.830588 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:01:16.830820 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:01:16.830826 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:01:16.831677 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:01:16.831678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:01:16.832830 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:01:16.832834 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:01:16.833410 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:01:16.833411 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:01:16.833591 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:16.834123 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:01:16.834131 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:01:16.835719 (+ 1588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:01:16.837584 (+ 1865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:01:16.837591 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:01:16.837593 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:01:16.837903 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:01:16.838133 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:01:16.838134 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:01:16.838134 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:01:16.838267 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:01:16.838585 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:01:16.838592 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:01:16.838776 (+ 184us) client_negotiation.cc:769] Sending connection context
0924 05:01:16.839549 (+ 773us) client_negotiation.cc:240] Negotiation successful
0924 05:01:16.839713 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3501,"thread_start_us":63,"threads_started":1}
I20260924 05:01:16.840569 13473 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33935
I20260924 05:01:16.840664 13473 heartbeater.cc:464] Registering TS with master...
I20260924 05:01:16.840853 13473 heartbeater.cc:511] Master 127.0.98.62:33935 requested a full tablet report, sending...
W20260924 05:01:16.841501 13473 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33935 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 05:01:16.841565 13473 heartbeater.cc:464] Registering TS with master...
I20260924 05:01:16.841617 13473 heartbeater.cc:511] Master 127.0.98.62:33935 requested a full tablet report, sending...
I20260924 05:01:16.841930 13473 heartbeater.cc:464] Registering TS with master...
I20260924 05:01:16.841995 13473 heartbeater.cc:511] Master 127.0.98.62:33935 requested a full tablet report, sending...
W20260924 05:01:16.842259 13473 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 05:01:16.925962 13476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:01:16.926288 13476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:01:16.926354 13476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:01:16.928046 13476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:01:16.928115 13476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:01:16.928159 13476 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 05:01:16.930392 13476 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:41535/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:44295
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=45683
--tserver_master_addrs=127.0.98.62:33935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13476
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:01:16.930671 13476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:01:16.931011 13476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:01:16.937536 13481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:01:16.937732 13476 server_base.cc:1034] running on GCE node
W20260924 05:01:16.937943 13482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:01:16.938122 13484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:01:16.938336 13476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:01:16.938654 13476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:01:16.939844 13476 hybrid_clock.cc:648] HybridClock initialized: now 1790226076939813 us; error 48 us; skew 500 ppm
Sep 24 05:01:16 dist-test-slave-70st krb5kdc[12145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226076, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:01:16.943295 13476 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 05:01:16.943858 13476 webserver.cc:460] Webserver started at http://127.0.98.3:45683/ using document root <none> and password file <none>
I20260924 05:01:16.944137 13476 fs_manager.cc:359] Metadata directory not provided
I20260924 05:01:16.944224 13476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 05:01:16 dist-test-slave-70st krb5kdc[12145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226076, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 05:01:16.965400 13476 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 05:01:16.966326 13493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:01:16.966531 13476 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 05:01:16.966629 13476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ts-2/wal
uuid: "9738e1af1b794ed6915de35b19fb335c"
format_stamp: "Formatted at 2026-09-24 05:01:16 on dist-test-slave-70st"
server_key: "62fa9103d83cbb2c9f38866043aa2256"
server_key_iv: "eac20e147bdb4f58daf059627868d84c"
server_key_version: "kuduclusterkey@0"
I20260924 05:01:16.967005 13476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:01:16.982865 13476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:01:16.985500 13476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:01:16.985702 13476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:01:16.985963 13476 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:01:16.986351 13476 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:01:16.986430 13476 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:01:16.986491 13476 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:01:16.986532 13476 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:01:16.993770 13476 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44295
I20260924 05:01:16.993851 13606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44295 every 8 connection(s)
I20260924 05:01:16.994230 13476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 05:01:16 dist-test-slave-70st krb5kdc[12145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226076, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:01:17.003618 13343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:16.994983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35825 (local address 127.0.98.62:33935)
0924 05:01:16.995040 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 05:01:16.995042 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:01:16.995342 (+ 300us) server_negotiation.cc:407] Connection header received
0924 05:01:16.995813 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:01:16.995814 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:01:16.995827 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:01:16.995884 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:01:16.996450 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:16.997198 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:16.998226 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:16.998542 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:16.998608 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:17.000975 (+ 2367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:01:17.000976 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:01:17.000976 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:01:17.000983 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:01:17.002378 (+ 1395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:01:17.002730 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:01:17.002731 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:01:17.002731 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:01:17.002794 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:01:17.003010 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:01:17.003011 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:01:17.003012 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:01:17.003139 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:01:17.003211 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 05:01:17.003440 (+ 229us) server_negotiation.cc:299] Negotiation successful
0924 05:01:17.003524 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 05:01:17.003671 13609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:16.995019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33935 (local address 127.0.98.3:35825)
0924 05:01:16.995188 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:16.995224 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 05:01:16.995709 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:01:16.995959 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:01:16.995963 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:01:16.996135 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:01:16.996338 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:01:16.996344 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:01:16.997287 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:01:16.997288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:01:16.998149 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:01:16.998152 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:01:16.998624 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:01:16.998625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:01:16.998726 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:16.999152 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:01:16.999160 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:01:17.000892 (+ 1732us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:01:17.002449 (+ 1557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:01:17.002454 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:01:17.002456 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:01:17.002660 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:01:17.002857 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:01:17.002858 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:01:17.002858 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:01:17.002955 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:01:17.003199 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:01:17.003202 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:01:17.003320 (+ 118us) client_negotiation.cc:769] Sending connection context
0924 05:01:17.003403 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 05:01:17.003515 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":92,"threads_started":1}
I20260924 05:01:17.004185 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 13476
I20260924 05:01:17.004349 13607 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33935
I20260924 05:01:17.004446 13607 heartbeater.cc:464] Registering TS with master...
I20260924 05:01:17.004630 13607 heartbeater.cc:511] Master 127.0.98.62:33935 requested a full tablet report, sending...
W20260924 05:01:17.005365 13607 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33935 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 05:01:17.005465 13607 heartbeater.cc:464] Registering TS with master...
I20260924 05:01:17.005528 13607 heartbeater.cc:511] Master 127.0.98.62:33935 requested a full tablet report, sending...
I20260924 05:01:17.005875 13607 heartbeater.cc:464] Registering TS with master...
I20260924 05:01:17.005960 13607 heartbeater.cc:511] Master 127.0.98.62:33935 requested a full tablet report, sending...
W20260924 05:01:17.006246 13607 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 05:01:17.014719 13343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:17.005497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40616 (local address 127.0.98.62:33935)
0924 05:01:17.005532 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 05:01:17.005534 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:01:17.005854 (+ 320us) server_negotiation.cc:407] Connection header received
0924 05:01:17.005873 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:01:17.005874 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:01:17.005890 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:01:17.005938 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:01:17.006807 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:17.007495 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:17.009477 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:17.009672 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:17.009741 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:17.011728 (+ 1987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:01:17.011729 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:01:17.011729 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:01:17.011736 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:01:17.013405 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:01:17.013800 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:01:17.013801 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:01:17.013802 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:01:17.013853 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:01:17.014042 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:01:17.014042 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:01:17.014043 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:01:17.014163 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:01:17.014308 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0924 05:01:17.014540 (+ 232us) server_negotiation.cc:299] Negotiation successful
0924 05:01:17.014620 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
Sep 24 05:01:17 dist-test-slave-70st krb5kdc[12145](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790226077, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:01:17.689952 13339 heartbeater.cc:464] Registering TS with master...
I20260924 05:01:17.690112 13339 heartbeater.cc:511] Master 127.0.98.62:33935 requested a full tablet report, sending...
I20260924 05:01:17.690541 13141 ts_manager.cc:194] Registered new tserver with Master: b69c8bdc782a4bd98b0de4c18c92b16d (127.0.98.1:44797)
I20260924 05:01:17.691526 13141 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:36807
I20260924 05:01:17.842411 13473 heartbeater.cc:464] Registering TS with master...
I20260924 05:01:17.842568 13473 heartbeater.cc:511] Master 127.0.98.62:33935 requested a full tablet report, sending...
I20260924 05:01:17.842875 13141 ts_manager.cc:194] Registered new tserver with Master: cc8a9a07a7c640df97e5d5bbefb75036 (127.0.98.2:45099)
I20260924 05:01:17.843392 13141 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53483
Sep 24 05:01:17 dist-test-slave-70st krb5kdc[12145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226077, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 05:01:18.006409 13607 heartbeater.cc:464] Registering TS with master...
I20260924 05:01:18.006563 13607 heartbeater.cc:511] Master 127.0.98.62:33935 requested a full tablet report, sending...
I20260924 05:01:18.006888 13141 ts_manager.cc:194] Registered new tserver with Master: 9738e1af1b794ed6915de35b19fb335c (127.0.98.3:44295)
I20260924 05:01:18.007423 13141 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35825
I20260924 05:01:18.014390 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 05:01:18.025336 13628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:18.015385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40618 (local address 127.0.98.62:33935)
0924 05:01:18.015501 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0924 05:01:18.015503 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:01:18.017602 (+ 2099us) server_negotiation.cc:407] Connection header received
0924 05:01:18.017722 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:01:18.017723 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:01:18.017738 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:01:18.017817 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:01:18.018518 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:18.019303 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:18.020345 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:18.020655 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:18.020728 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:18.022204 (+ 1476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:01:18.022206 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:01:18.022206 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:01:18.022214 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:01:18.024037 (+ 1823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:01:18.024429 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:01:18.024431 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:01:18.024431 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:01:18.024497 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:01:18.024719 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:01:18.024720 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:01:18.024721 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:01:18.024878 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:01:18.024985 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 05:01:18.025124 (+ 139us) server_negotiation.cc:299] Negotiation successful
0924 05:01:18.025211 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":64,"threads_started":1}
I20260924 05:01:18.027664 13141 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40618:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 05:01:18.029894 13141 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 05:01:18.038880 13637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:18.033381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44797 (local address 127.0.0.1:51530)
0924 05:01:18.034176 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:18.034187 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 05:01:18.034257 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:01:18.034565 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:01:18.034568 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:01:18.034570 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:01:18.034647 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:01:18.034649 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:01:18.036084 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:01:18.036085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:01:18.037363 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:01:18.037365 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:01:18.038651 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:01:18.038652 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:01:18.038709 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:18.038713 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 05:01:18.038753 (+ 40us) client_negotiation.cc:240] Negotiation successful
0924 05:01:18.038790 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":776,"mutex_wait_us":62,"thread_start_us":329,"threads_started":1}
I20260924 05:01:18.039161 13635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:18.033526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51530 (local address 127.0.98.1:44797)
0924 05:01:18.033659 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 05:01:18.033664 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 05:01:18.034325 (+ 661us) server_negotiation.cc:407] Connection header received
0924 05:01:18.034392 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:01:18.034396 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:01:18.034411 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:01:18.034478 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:01:18.034716 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:18.036014 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:18.037490 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:18.038487 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:18.038548 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:18.038620 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 05:01:18.039054 (+ 434us) server_negotiation.cc:299] Negotiation successful
0924 05:01:18.039093 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":70,"threads_started":1}
I20260924 05:01:18.039656 13636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:18.033121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44295 (local address 127.0.0.1:59676)
0924 05:01:18.033751 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:18.033790 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 05:01:18.034098 (+ 308us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:01:18.034693 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:01:18.034694 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:01:18.034696 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:01:18.034761 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:01:18.034763 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:01:18.036048 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:01:18.036049 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:01:18.037396 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:01:18.037400 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:01:18.038485 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:01:18.038486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:01:18.038568 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:18.038582 (+ 14us) client_negotiation.cc:769] Sending connection context
0924 05:01:18.038661 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 05:01:18.038730 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"mutex_wait_us":55,"thread_start_us":90,"threads_started":1}
I20260924 05:01:18.038964 13634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:18.033205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59676 (local address 127.0.98.3:44295)
0924 05:01:18.034447 (+ 1242us) server_negotiation.cc:206] Beginning negotiation
0924 05:01:18.034450 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:01:18.034462 (+ 12us) server_negotiation.cc:407] Connection header received
0924 05:01:18.034499 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:01:18.034501 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:01:18.034517 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:01:18.034603 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:01:18.034855 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:18.035942 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:18.037546 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:18.038281 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:18.038362 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:18.038443 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 05:01:18.038841 (+ 398us) server_negotiation.cc:299] Negotiation successful
0924 05:01:18.038882 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1208,"thread_start_us":56,"threads_started":1}
I20260924 05:01:18.040055 13273 tablet_service.cc:1467] Processing CreateTablet for tablet e48851d9f53249caba947326a424b8fa (DEFAULT_TABLE table=test-table [id=2af985f400a84ae483911526f7975f42]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:01:18.040463 13273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e48851d9f53249caba947326a424b8fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:01:18.040719 13541 tablet_service.cc:1467] Processing CreateTablet for tablet e48851d9f53249caba947326a424b8fa (DEFAULT_TABLE table=test-table [id=2af985f400a84ae483911526f7975f42]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:01:18.041006 13541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e48851d9f53249caba947326a424b8fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:01:18.043373 13638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:18.034035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49252 (local address 127.0.98.2:45099)
0924 05:01:18.034371 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0924 05:01:18.034375 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 05:01:18.039366 (+ 4991us) server_negotiation.cc:407] Connection header received
0924 05:01:18.039473 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:01:18.039475 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:01:18.039490 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:01:18.039539 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:01:18.039719 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:18.041032 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:18.042244 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:18.042932 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:18.042979 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:18.043047 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 05:01:18.043241 (+ 194us) server_negotiation.cc:299] Negotiation successful
0924 05:01:18.043284 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":56,"threads_started":1}
I20260924 05:01:18.043610 13637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:18.033830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45099 (local address 127.0.0.1:49252)
0924 05:01:18.039295 (+ 5465us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:18.039304 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:01:18.039395 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:01:18.039629 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:01:18.039629 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:01:18.039630 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:01:18.039659 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:01:18.039660 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:01:18.041106 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:01:18.041106 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:01:18.042157 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:01:18.042160 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:01:18.043101 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:01:18.043102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:01:18.043158 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:18.043162 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 05:01:18.043511 (+ 349us) client_negotiation.cc:240] Negotiation successful
0924 05:01:18.043551 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5455,"thread_start_us":64,"threads_started":1}
I20260924 05:01:18.044358 13639 tablet_bootstrap.cc:492] T e48851d9f53249caba947326a424b8fa P b69c8bdc782a4bd98b0de4c18c92b16d: Bootstrap starting.
I20260924 05:01:18.044698 13407 tablet_service.cc:1467] Processing CreateTablet for tablet e48851d9f53249caba947326a424b8fa (DEFAULT_TABLE table=test-table [id=2af985f400a84ae483911526f7975f42]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:01:18.045014 13639 tablet_bootstrap.cc:654] T e48851d9f53249caba947326a424b8fa P b69c8bdc782a4bd98b0de4c18c92b16d: Neither blocks nor log segments found. Creating new log.
I20260924 05:01:18.045048 13407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e48851d9f53249caba947326a424b8fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:01:18.045295 13639 log.cc:824] T e48851d9f53249caba947326a424b8fa P b69c8bdc782a4bd98b0de4c18c92b16d: Log is configured to *not* fsync() on all Append() calls
I20260924 05:01:18.048709 13641 tablet_bootstrap.cc:492] T e48851d9f53249caba947326a424b8fa P cc8a9a07a7c640df97e5d5bbefb75036: Bootstrap starting.
I20260924 05:01:18.049348 13641 tablet_bootstrap.cc:654] T e48851d9f53249caba947326a424b8fa P cc8a9a07a7c640df97e5d5bbefb75036: Neither blocks nor log segments found. Creating new log.
I20260924 05:01:18.049759 13641 log.cc:824] T e48851d9f53249caba947326a424b8fa P cc8a9a07a7c640df97e5d5bbefb75036: Log is configured to *not* fsync() on all Append() calls
I20260924 05:01:18.050235 13643 tablet_bootstrap.cc:492] T e48851d9f53249caba947326a424b8fa P 9738e1af1b794ed6915de35b19fb335c: Bootstrap starting.
I20260924 05:01:18.050906 13643 tablet_bootstrap.cc:654] T e48851d9f53249caba947326a424b8fa P 9738e1af1b794ed6915de35b19fb335c: Neither blocks nor log segments found. Creating new log.
I20260924 05:01:18.051095 13639 tablet_bootstrap.cc:492] T e48851d9f53249caba947326a424b8fa P b69c8bdc782a4bd98b0de4c18c92b16d: No bootstrap required, opened a new log
I20260924 05:01:18.051177 13639 ts_tablet_manager.cc:1397] T e48851d9f53249caba947326a424b8fa P b69c8bdc782a4bd98b0de4c18c92b16d: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260924 05:01:18.051242 13643 log.cc:824] T e48851d9f53249caba947326a424b8fa P 9738e1af1b794ed6915de35b19fb335c: Log is configured to *not* fsync() on all Append() calls
I20260924 05:01:18.051888 13643 tablet_bootstrap.cc:492] T e48851d9f53249caba947326a424b8fa P 9738e1af1b794ed6915de35b19fb335c: No bootstrap required, opened a new log
I20260924 05:01:18.051937 13643 ts_tablet_manager.cc:1397] T e48851d9f53249caba947326a424b8fa P 9738e1af1b794ed6915de35b19fb335c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 05:01:18.054293 13639 raft_consensus.cc:348] T e48851d9f53249caba947326a424b8fa P b69c8bdc782a4bd98b0de4c18c92b16d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9738e1af1b794ed6915de35b19fb335c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44295 } } peers { permanent_uuid: "b69c8bdc782a4bd98b0de4c18c92b16d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44797 } } peers { permanent_uuid: "cc8a9a07a7c640df97e5d5bbefb75036" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45099 } }
I20260924 05:01:18.054581 13639 raft_consensus.cc:374] T e48851d9f53249caba947326a424b8fa P b69c8bdc782a4bd98b0de4c18c92b16d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:01:18.054674 13639 raft_consensus.cc:729] T e48851d9f53249caba947326a424b8fa P b69c8bdc782a4bd98b0de4c18c92b16d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b69c8bdc782a4bd98b0de4c18c92b16d, State: Initialized, Role: FOLLOWER
I20260924 05:01:18.054780 13643 raft_consensus.cc:348] T e48851d9f53249caba947326a424b8fa P 9738e1af1b794ed6915de35b19fb335c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9738e1af1b794ed6915de35b19fb335c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44295 } } peers { permanent_uuid: "b69c8bdc782a4bd98b0de4c18c92b16d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44797 } } peers { permanent_uuid: "cc8a9a07a7c640df97e5d5bbefb75036" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45099 } }
I20260924 05:01:18.054931 13643 raft_consensus.cc:374] T e48851d9f53249caba947326a424b8fa P 9738e1af1b794ed6915de35b19fb335c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:01:18.054978 13643 raft_consensus.cc:729] T e48851d9f53249caba947326a424b8fa P 9738e1af1b794ed6915de35b19fb335c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9738e1af1b794ed6915de35b19fb335c, State: Initialized, Role: FOLLOWER
I20260924 05:01:18.054896 13639 consensus_queue.cc:260] T e48851d9f53249caba947326a424b8fa P b69c8bdc782a4bd98b0de4c18c92b16d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9738e1af1b794ed6915de35b19fb335c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44295 } } peers { permanent_uuid: "b69c8bdc782a4bd98b0de4c18c92b16d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44797 } } peers { permanent_uuid: "cc8a9a07a7c640df97e5d5bbefb75036" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45099 } }
I20260924 05:01:18.055217 13639 ts_tablet_manager.cc:1428] T e48851d9f53249caba947326a424b8fa P b69c8bdc782a4bd98b0de4c18c92b16d: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260924 05:01:18.055207 13643 consensus_queue.cc:260] T e48851d9f53249caba947326a424b8fa P 9738e1af1b794ed6915de35b19fb335c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9738e1af1b794ed6915de35b19fb335c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44295 } } peers { permanent_uuid: "b69c8bdc782a4bd98b0de4c18c92b16d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44797 } } peers { permanent_uuid: "cc8a9a07a7c640df97e5d5bbefb75036" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45099 } }
I20260924 05:01:18.055454 13643 ts_tablet_manager.cc:1428] T e48851d9f53249caba947326a424b8fa P 9738e1af1b794ed6915de35b19fb335c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 05:01:18.055455 13339 heartbeater.cc:503] Master 127.0.98.62:33935 was elected leader, sending a full tablet report...
I20260924 05:01:18.055605 13607 heartbeater.cc:503] Master 127.0.98.62:33935 was elected leader, sending a full tablet report...
I20260924 05:01:18.058161 13641 tablet_bootstrap.cc:492] T e48851d9f53249caba947326a424b8fa P cc8a9a07a7c640df97e5d5bbefb75036: No bootstrap required, opened a new log
I20260924 05:01:18.058244 13641 ts_tablet_manager.cc:1397] T e48851d9f53249caba947326a424b8fa P cc8a9a07a7c640df97e5d5bbefb75036: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s
I20260924 05:01:18.060753 13641 raft_consensus.cc:348] T e48851d9f53249caba947326a424b8fa P cc8a9a07a7c640df97e5d5bbefb75036 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9738e1af1b794ed6915de35b19fb335c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44295 } } peers { permanent_uuid: "b69c8bdc782a4bd98b0de4c18c92b16d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44797 } } peers { permanent_uuid: "cc8a9a07a7c640df97e5d5bbefb75036" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45099 } }
I20260924 05:01:18.060904 13641 raft_consensus.cc:374] T e48851d9f53249caba947326a424b8fa P cc8a9a07a7c640df97e5d5bbefb75036 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:01:18.060932 13641 raft_consensus.cc:729] T e48851d9f53249caba947326a424b8fa P cc8a9a07a7c640df97e5d5bbefb75036 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc8a9a07a7c640df97e5d5bbefb75036, State: Initialized, Role: FOLLOWER
I20260924 05:01:18.061048 13641 consensus_queue.cc:260] T e48851d9f53249caba947326a424b8fa P cc8a9a07a7c640df97e5d5bbefb75036 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9738e1af1b794ed6915de35b19fb335c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44295 } } peers { permanent_uuid: "b69c8bdc782a4bd98b0de4c18c92b16d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44797 } } peers { permanent_uuid: "cc8a9a07a7c640df97e5d5bbefb75036" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45099 } }
I20260924 05:01:18.061347 13473 heartbeater.cc:503] Master 127.0.98.62:33935 was elected leader, sending a full tablet report...
I20260924 05:01:18.061905 13641 ts_tablet_manager.cc:1428] T e48851d9f53249caba947326a424b8fa P cc8a9a07a7c640df97e5d5bbefb75036: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
W20260924 05:01:18.075965 13474 tablet.cc:2369] T e48851d9f53249caba947326a424b8fa P cc8a9a07a7c640df97e5d5bbefb75036: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 05:01:18 dist-test-slave-70st krb5kdc[12145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226077, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260924 05:01:18.170691 13340 tablet.cc:2369] T e48851d9f53249caba947326a424b8fa P b69c8bdc782a4bd98b0de4c18c92b16d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 05:01:18.246093 13608 tablet.cc:2369] T e48851d9f53249caba947326a424b8fa P 9738e1af1b794ed6915de35b19fb335c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 05:01:18.342051 13646 raft_consensus.cc:482] T e48851d9f53249caba947326a424b8fa P 9738e1af1b794ed6915de35b19fb335c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 05:01:18.342193 13646 raft_consensus.cc:504] T e48851d9f53249caba947326a424b8fa P 9738e1af1b794ed6915de35b19fb335c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9738e1af1b794ed6915de35b19fb335c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44295 } } peers { permanent_uuid: "b69c8bdc782a4bd98b0de4c18c92b16d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44797 } } peers { permanent_uuid: "cc8a9a07a7c640df97e5d5bbefb75036" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45099 } }
I20260924 05:01:18.342592 13646 leader_election.cc:288] T e48851d9f53249caba947326a424b8fa P 9738e1af1b794ed6915de35b19fb335c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b69c8bdc782a4bd98b0de4c18c92b16d (127.0.98.1:44797), cc8a9a07a7c640df97e5d5bbefb75036 (127.0.98.2:45099)
I20260924 05:01:18.345602 13635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:18.342817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58523 (local address 127.0.98.1:44797)
0924 05:01:18.342881 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 05:01:18.342884 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:01:18.343110 (+ 226us) server_negotiation.cc:407] Connection header received
0924 05:01:18.343210 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:01:18.343211 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:01:18.343224 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:01:18.343268 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:01:18.343422 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:18.343880 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:18.344828 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:18.345225 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:18.345285 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:18.345295 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 05:01:18.345479 (+ 184us) server_negotiation.cc:299] Negotiation successful
0924 05:01:18.345534 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 05:01:18.345613 13658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:18.342752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44797 (local address 127.0.98.3:58523)
0924 05:01:18.343045 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:18.343056 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 05:01:18.343132 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:01:18.343270 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:01:18.343271 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:01:18.343272 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:01:18.343357 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:01:18.343359 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:01:18.343948 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:01:18.343949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:01:18.344757 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:01:18.344760 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:01:18.345291 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:01:18.345292 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:01:18.345361 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:18.345365 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 05:01:18.345464 (+ 99us) client_negotiation.cc:240] Negotiation successful
0924 05:01:18.345511 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":80,"threads_started":1}
I20260924 05:01:18.346241 13638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:18.343036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42777 (local address 127.0.98.2:45099)
0924 05:01:18.343082 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 05:01:18.343083 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:01:18.343097 (+ 14us) server_negotiation.cc:407] Connection header received
0924 05:01:18.343193 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:01:18.343194 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:01:18.343206 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:01:18.343253 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:01:18.343459 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:18.344158 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:18.345453 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:18.345934 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:18.345988 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:18.345996 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 05:01:18.346152 (+ 156us) server_negotiation.cc:299] Negotiation successful
0924 05:01:18.346191 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 05:01:18.346252 13657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:18.342939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45099 (local address 127.0.98.3:42777)
0924 05:01:18.343045 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:18.343056 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 05:01:18.343110 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:01:18.343303 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:01:18.343305 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:01:18.343307 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:01:18.343386 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:01:18.343388 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:01:18.344254 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:01:18.344254 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:01:18.345328 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:01:18.345331 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:01:18.346017 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:01:18.346018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:01:18.346087 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:18.346091 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 05:01:18.346146 (+ 55us) client_negotiation.cc:240] Negotiation successful
0924 05:01:18.346179 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":56,"threads_started":1}
I20260924 05:01:18.346462 13293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e48851d9f53249caba947326a424b8fa" candidate_uuid: "9738e1af1b794ed6915de35b19fb335c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b69c8bdc782a4bd98b0de4c18c92b16d" is_pre_election: true
I20260924 05:01:18.346622 13293 raft_consensus.cc:2454] T e48851d9f53249caba947326a424b8fa P b69c8bdc782a4bd98b0de4c18c92b16d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9738e1af1b794ed6915de35b19fb335c in term 0.
I20260924 05:01:18.346849 13495 leader_election.cc:302] T e48851d9f53249caba947326a424b8fa P 9738e1af1b794ed6915de35b19fb335c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9738e1af1b794ed6915de35b19fb335c, b69c8bdc782a4bd98b0de4c18c92b16d; no voters:
I20260924 05:01:18.346972 13646 raft_consensus.cc:2781] T e48851d9f53249caba947326a424b8fa P 9738e1af1b794ed6915de35b19fb335c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 05:01:18.347047 13646 raft_consensus.cc:482] T e48851d9f53249caba947326a424b8fa P 9738e1af1b794ed6915de35b19fb335c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 05:01:18.347074 13646 raft_consensus.cc:3037] T e48851d9f53249caba947326a424b8fa P 9738e1af1b794ed6915de35b19fb335c [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:01:18.347147 13427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e48851d9f53249caba947326a424b8fa" candidate_uuid: "9738e1af1b794ed6915de35b19fb335c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc8a9a07a7c640df97e5d5bbefb75036" is_pre_election: true
I20260924 05:01:18.347268 13427 raft_consensus.cc:2454] T e48851d9f53249caba947326a424b8fa P cc8a9a07a7c640df97e5d5bbefb75036 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9738e1af1b794ed6915de35b19fb335c in term 0.
I20260924 05:01:18.347961 13646 raft_consensus.cc:504] T e48851d9f53249caba947326a424b8fa P 9738e1af1b794ed6915de35b19fb335c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9738e1af1b794ed6915de35b19fb335c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44295 } } peers { permanent_uuid: "b69c8bdc782a4bd98b0de4c18c92b16d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44797 } } peers { permanent_uuid: "cc8a9a07a7c640df97e5d5bbefb75036" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45099 } }
I20260924 05:01:18.348080 13646 leader_election.cc:288] T e48851d9f53249caba947326a424b8fa P 9738e1af1b794ed6915de35b19fb335c [CANDIDATE]: Term 1 election: Requested vote from peers b69c8bdc782a4bd98b0de4c18c92b16d (127.0.98.1:44797), cc8a9a07a7c640df97e5d5bbefb75036 (127.0.98.2:45099)
I20260924 05:01:18.348227 13427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e48851d9f53249caba947326a424b8fa" candidate_uuid: "9738e1af1b794ed6915de35b19fb335c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc8a9a07a7c640df97e5d5bbefb75036"
I20260924 05:01:18.348295 13427 raft_consensus.cc:3037] T e48851d9f53249caba947326a424b8fa P cc8a9a07a7c640df97e5d5bbefb75036 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:01:18.348974 13427 raft_consensus.cc:2454] T e48851d9f53249caba947326a424b8fa P cc8a9a07a7c640df97e5d5bbefb75036 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9738e1af1b794ed6915de35b19fb335c in term 1.
I20260924 05:01:18.349073 13293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e48851d9f53249caba947326a424b8fa" candidate_uuid: "9738e1af1b794ed6915de35b19fb335c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b69c8bdc782a4bd98b0de4c18c92b16d"
I20260924 05:01:18.349162 13495 leader_election.cc:302] T e48851d9f53249caba947326a424b8fa P 9738e1af1b794ed6915de35b19fb335c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9738e1af1b794ed6915de35b19fb335c, cc8a9a07a7c640df97e5d5bbefb75036; no voters:
I20260924 05:01:18.349150 13293 raft_consensus.cc:3037] T e48851d9f53249caba947326a424b8fa P b69c8bdc782a4bd98b0de4c18c92b16d [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:01:18.349246 13646 raft_consensus.cc:2781] T e48851d9f53249caba947326a424b8fa P 9738e1af1b794ed6915de35b19fb335c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:01:18.349386 13646 raft_consensus.cc:686] T e48851d9f53249caba947326a424b8fa P 9738e1af1b794ed6915de35b19fb335c [term 1 LEADER]: Becoming Leader. State: Replica: 9738e1af1b794ed6915de35b19fb335c, State: Running, Role: LEADER
I20260924 05:01:18.349505 13646 consensus_queue.cc:237] T e48851d9f53249caba947326a424b8fa P 9738e1af1b794ed6915de35b19fb335c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9738e1af1b794ed6915de35b19fb335c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44295 } } peers { permanent_uuid: "b69c8bdc782a4bd98b0de4c18c92b16d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44797 } } peers { permanent_uuid: "cc8a9a07a7c640df97e5d5bbefb75036" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45099 } }
I20260924 05:01:18.349941 13293 raft_consensus.cc:2454] T e48851d9f53249caba947326a424b8fa P b69c8bdc782a4bd98b0de4c18c92b16d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9738e1af1b794ed6915de35b19fb335c in term 1.
I20260924 05:01:18.350411 13141 catalog_manager.cc:5495] T e48851d9f53249caba947326a424b8fa P 9738e1af1b794ed6915de35b19fb335c reported cstate change: term changed from 0 to 1, leader changed from <none> to 9738e1af1b794ed6915de35b19fb335c (127.0.98.3). New cstate: current_term: 1 leader_uuid: "9738e1af1b794ed6915de35b19fb335c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9738e1af1b794ed6915de35b19fb335c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44295 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b69c8bdc782a4bd98b0de4c18c92b16d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc8a9a07a7c640df97e5d5bbefb75036" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45099 } health_report { overall_health: UNKNOWN } } }
I20260924 05:01:18.390058 13634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:18.387370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59690 (local address 127.0.98.3:44295)
0924 05:01:18.387421 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 05:01:18.387423 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:01:18.387430 (+ 7us) server_negotiation.cc:407] Connection header received
0924 05:01:18.387465 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:01:18.387465 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:01:18.387484 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:01:18.387547 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:01:18.387682 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:18.388153 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:18.389213 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:18.389449 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:18.389506 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:18.389716 (+ 210us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:01:18.389854 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:01:18.389937 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 05:01:18.389956 (+ 19us) server_negotiation.cc:299] Negotiation successful
0924 05:01:18.389981 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 05:01:18.391760 13427 raft_consensus.cc:1261] T e48851d9f53249caba947326a424b8fa P cc8a9a07a7c640df97e5d5bbefb75036 [term 1 FOLLOWER]: Refusing update from remote peer 9738e1af1b794ed6915de35b19fb335c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:01:18.391760 13293 raft_consensus.cc:1261] T e48851d9f53249caba947326a424b8fa P b69c8bdc782a4bd98b0de4c18c92b16d [term 1 FOLLOWER]: Refusing update from remote peer 9738e1af1b794ed6915de35b19fb335c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:01:18.392103 13646 consensus_queue.cc:1035] T e48851d9f53249caba947326a424b8fa P 9738e1af1b794ed6915de35b19fb335c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b69c8bdc782a4bd98b0de4c18c92b16d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:01:18.392238 13659 consensus_queue.cc:1035] T e48851d9f53249caba947326a424b8fa P 9738e1af1b794ed6915de35b19fb335c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc8a9a07a7c640df97e5d5bbefb75036" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:01:18.394753 13662 mvcc.cc:205] Tried to move back new op lower bound from 7332766017090039808 to 7332766016920276992. Current Snapshot: MvccSnapshot[applied={T|T < 7332766017090039808}]
I20260924 05:01:18.396008 13663 mvcc.cc:205] Tried to move back new op lower bound from 7332766017090039808 to 7332766016920276992. Current Snapshot: MvccSnapshot[applied={T|T < 7332766017090039808 or (T in {7332766017090039808})}]
I20260924 05:01:18.399439 13140 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40618:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:01:18.399541 13140 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40618:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:01:18.400287 13140 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4ebc24cba59046c5be83914718165fbc: Sending DeleteTablet for 3 replicas of tablet e48851d9f53249caba947326a424b8fa
I20260924 05:01:18.400722 13273 tablet_service.cc:1514] Processing DeleteTablet for tablet e48851d9f53249caba947326a424b8fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:01:18 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51530
I20260924 05:01:18.400826 13541 tablet_service.cc:1514] Processing DeleteTablet for tablet e48851d9f53249caba947326a424b8fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:01:18 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59676
I20260924 05:01:18.401057 13407 tablet_service.cc:1514] Processing DeleteTablet for tablet e48851d9f53249caba947326a424b8fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:01:18 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49252
I20260924 05:01:18.401281 13673 tablet_replica.cc:330] T e48851d9f53249caba947326a424b8fa P cc8a9a07a7c640df97e5d5bbefb75036: stopping tablet replica
I20260924 05:01:18.401448 13673 raft_consensus.cc:2229] T e48851d9f53249caba947326a424b8fa P cc8a9a07a7c640df97e5d5bbefb75036 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:01:18.401597 13673 raft_consensus.cc:2258] T e48851d9f53249caba947326a424b8fa P cc8a9a07a7c640df97e5d5bbefb75036 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:01:18.401659 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 13179
I20260924 05:01:18.402935 13673 ts_tablet_manager.cc:1905] T e48851d9f53249caba947326a424b8fa P cc8a9a07a7c640df97e5d5bbefb75036: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:01:18.404647 13673 ts_tablet_manager.cc:1918] T e48851d9f53249caba947326a424b8fa P cc8a9a07a7c640df97e5d5bbefb75036: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 05:01:18.404722 13673 log.cc:1196] T e48851d9f53249caba947326a424b8fa P cc8a9a07a7c640df97e5d5bbefb75036: Deleting WAL directory at /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ts-1/wal/wals/e48851d9f53249caba947326a424b8fa
I20260924 05:01:18.405006 13673 ts_tablet_manager.cc:1939] T e48851d9f53249caba947326a424b8fa P cc8a9a07a7c640df97e5d5bbefb75036: Deleting consensus metadata
I20260924 05:01:18.405505 13672 tablet_replica.cc:330] T e48851d9f53249caba947326a424b8fa P 9738e1af1b794ed6915de35b19fb335c: stopping tablet replica
I20260924 05:01:18.405560 13126 catalog_manager.cc:4841] TS cc8a9a07a7c640df97e5d5bbefb75036 (127.0.98.2:45099): tablet e48851d9f53249caba947326a424b8fa (table test-table [id=2af985f400a84ae483911526f7975f42]) successfully deleted
I20260924 05:01:18.405668 13672 raft_consensus.cc:2229] T e48851d9f53249caba947326a424b8fa P 9738e1af1b794ed6915de35b19fb335c [term 1 LEADER]: Raft consensus shutting down.
I20260924 05:01:18.405798 13672 raft_consensus.cc:2258] T e48851d9f53249caba947326a424b8fa P 9738e1af1b794ed6915de35b19fb335c [term 1 FOLLOWER]: Raft consensus is shut down!
W20260924 05:01:18.409036 13126 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44797 (error 108)
I20260924 05:01:18.409282 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 13341
I20260924 05:01:18.409958 13633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:18.409525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44797 (local address 127.0.0.1:51546)
0924 05:01:18.409852 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:18.409914 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":291}
I20260924 05:01:18.410238 13672 ts_tablet_manager.cc:1905] T e48851d9f53249caba947326a424b8fa P 9738e1af1b794ed6915de35b19fb335c: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 05:01:18.410298 13126 catalog_manager.cc:4584] TS b69c8bdc782a4bd98b0de4c18c92b16d (127.0.98.1:44797): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e48851d9f53249caba947326a424b8fa: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44797: connect: Connection refused (error 111)
I20260924 05:01:18.412029 13672 ts_tablet_manager.cc:1918] T e48851d9f53249caba947326a424b8fa P 9738e1af1b794ed6915de35b19fb335c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 05:01:18.412091 13672 log.cc:1196] T e48851d9f53249caba947326a424b8fa P 9738e1af1b794ed6915de35b19fb335c: Deleting WAL directory at /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790225929848825-392-0/minicluster-data/ts-2/wal/wals/e48851d9f53249caba947326a424b8fa
I20260924 05:01:18.412307 13672 ts_tablet_manager.cc:1939] T e48851d9f53249caba947326a424b8fa P 9738e1af1b794ed6915de35b19fb335c: Deleting consensus metadata
I20260924 05:01:18.412959 13126 catalog_manager.cc:4841] TS 9738e1af1b794ed6915de35b19fb335c (127.0.98.3:44295): tablet e48851d9f53249caba947326a424b8fa (table test-table [id=2af985f400a84ae483911526f7975f42]) successfully deleted
I20260924 05:01:18.416802 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 13476
I20260924 05:01:18.420909 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 13108
I20260924 05:01:18.426972 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 05:01:18.761216 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 05:01:18.761315 392 mini_ranger.cc:66] Stopping Ranger...
I20260924 05:01:19.095607 392 mini_ranger.cc:68] Stopped Ranger
2026-09-24 05:01:19.095 UTC [12175] LOG: received smart shutdown request
2026-09-24 05:01:19.098 UTC [12175] LOG: background worker "logical replication launcher" (PID 12184) exited with exit code 1
2026-09-24 05:01:19.099 UTC [12179] LOG: shutting down
2026-09-24 05:01:19.259 UTC [12175] LOG: database system is shut down
2026-09-24T05:01:19Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (46712 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225929848825-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:01:19 dist-test-slave-70st krb5kdc[13683](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:01:19 dist-test-slave-70st krb5kdc[13683](info): set up 2 sockets
Sep 24 05:01:19 dist-test-slave-70st krb5kdc[13683](info): commencing operation
krb5kdc: starting...
W20260924 05:01:20.309037 392 mini_kdc.cc:121] Time spent starting KDC: real 0.944s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225929848825-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225929848825-392-0/krb5kdc/test-user.keytab.
Sep 24 05:01:20 dist-test-slave-70st krb5kdc[13683](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226080, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:01:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:01:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 05:01:20.418255 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225929848825-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225929848825-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225929848825-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40217
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40217,127.0.98.61:37191,127.0.98.60:45693
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:01:20.495426 13696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:01:20.495635 13696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:01:20.495664 13696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:01:20.496781 13696 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:01:20.496824 13696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:01:20.496841 13696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:01:20.496856 13696 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:01:20.496870 13696 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:01:20.498342 13696 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225929848825-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225929848825-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40217,127.0.98.61:37191,127.0.98.60:45693
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40217
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13696
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225929848825-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:01:20.498580 13696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:01:20.498777 13696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:01:20.503257 13702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:01:20.503305 13704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:01:20.503293 13701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:01:20.503501 13696 server_base.cc:1034] running on GCE node
I20260924 05:01:20.503707 13696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:01:20.503950 13696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:01:20.505086 13696 hybrid_clock.cc:648] HybridClock initialized: now 1790226080505074 us; error 28 us; skew 500 ppm
Sep 24 05:01:20 dist-test-slave-70st krb5kdc[13683](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226080, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:01:20.516021 13696 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 05:01:20.516474 13696 webserver.cc:460] Webserver started at http://127.0.98.62:35617/ using document root <none> and password file <none>
I20260924 05:01:20.516667 13696 fs_manager.cc:359] Metadata directory not provided
I20260924 05:01:20.516716 13696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:01:20.516839 13696 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:01:20.517922 13696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225929848825-392-0/minicluster-data/master-0/data/instance:
uuid: "9ea6743375da4d068e0f234d2f09636b"
format_stamp: "Formatted at 2026-09-24 05:01:20 on dist-test-slave-70st"
server_key: "cfed098a412b727abb1b21bfdbec3246"
server_key_iv: "e6d0b0c21e235f4fab66349e0b915782"
server_key_version: "encryptionkey@0"
I20260924 05:01:20.518250 13696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225929848825-392-0/minicluster-data/master-0/wal/instance:
uuid: "9ea6743375da4d068e0f234d2f09636b"
format_stamp: "Formatted at 2026-09-24 05:01:20 on dist-test-slave-70st"
server_key: "cfed098a412b727abb1b21bfdbec3246"
server_key_iv: "e6d0b0c21e235f4fab66349e0b915782"
server_key_version: "encryptionkey@0"
I20260924 05:01:20.519428 13696 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 05:01:20.520068 13711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:01:20.520210 13696 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 05:01:20.520287 13696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225929848825-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225929848825-392-0/minicluster-data/master-0/wal
uuid: "9ea6743375da4d068e0f234d2f09636b"
format_stamp: "Formatted at 2026-09-24 05:01:20 on dist-test-slave-70st"
server_key: "cfed098a412b727abb1b21bfdbec3246"
server_key_iv: "e6d0b0c21e235f4fab66349e0b915782"
server_key_version: "encryptionkey@0"
I20260924 05:01:20.520345 13696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225929848825-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225929848825-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225929848825-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:01:20.536821 13696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:01:20.538587 13696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:01:20.538720 13696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:01:20.542480 13696 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40217
I20260924 05:01:20.542511 13763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40217 every 8 connection(s)
I20260924 05:01:20.542827 13696 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:01:20.543630 13764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:01:20.545138 13764 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40217 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 05:01:20 dist-test-slave-70st krb5kdc[13683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226080, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:01:20.552539 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 13696
I20260924 05:01:20.552650 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225929848825-392-0/minicluster-data/master-0/wal/instance
I20260924 05:01:20.552817 392 external_mini_cluster.cc:1428] Setting key e5c723a06b01585091310b95f1c6186c
I20260924 05:01:20.553520 13767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:20.545782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58556 (local address 127.0.98.62:40217)
0924 05:01:20.545935 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0924 05:01:20.545942 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 05:01:20.546037 (+ 95us) server_negotiation.cc:407] Connection header received
0924 05:01:20.546507 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:01:20.546509 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:01:20.546524 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:01:20.546600 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:01:20.547078 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:20.547860 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:20.548655 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:20.548965 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:20.549042 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:20.550808 (+ 1766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:01:20.550815 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:01:20.550815 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:01:20.550821 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:01:20.552146 (+ 1325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:01:20.552433 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:01:20.552435 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:01:20.552435 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:01:20.552506 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:01:20.552723 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:01:20.552724 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:01:20.552724 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:01:20.552949 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:01:20.553009 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 05:01:20.553190 (+ 181us) server_negotiation.cc:299] Negotiation successful
0924 05:01:20.553296 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":16,"server-negotiator.queue_time_us":116,"thread_start_us":82,"threads_started":1}
I20260924 05:01:20.553520 13766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:20.545766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40217 (local address 127.0.0.1:58556)
0924 05:01:20.545929 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:20.545967 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 05:01:20.546394 (+ 427us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:01:20.546615 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:01:20.546617 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:01:20.546851 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:01:20.547015 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:01:20.547019 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:01:20.547930 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:01:20.547931 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:01:20.548587 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:01:20.548589 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:01:20.549026 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:01:20.549026 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:01:20.549077 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:20.549487 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:01:20.549494 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:01:20.550738 (+ 1244us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:01:20.552205 (+ 1467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:01:20.552209 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:01:20.552210 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:01:20.552368 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:01:20.552558 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:01:20.552559 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:01:20.552559 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:01:20.552662 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:01:20.552999 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:01:20.553008 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:01:20.553079 (+ 71us) client_negotiation.cc:769] Sending connection context
0924 05:01:20.553173 (+ 94us) client_negotiation.cc:240] Negotiation successful
0924 05:01:20.553354 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"mutex_wait_us":47,"thread_start_us":62,"threads_started":1}
I20260924 05:01:20.554602 13764 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37191 } has no permanent_uuid. Determining permanent_uuid...
I20260924 05:01:20.554896 13766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:20.554781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37191 (local address 127.0.0.1:52420)
0924 05:01:20.554825 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:20.554871 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260924 05:01:20.555027 13713 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37191: connect: Connection refused (error 111)
I20260924 05:01:20.555377 13766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:20.555286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37191 (local address 127.0.0.1:52434)
0924 05:01:20.555329 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:20.555354 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260924 05:01:20.555469 13764 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:37191: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37191: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225929848825-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225929848825-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260924 05:01:20.575742 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225929848825-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225929848825-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225929848825-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225929848825-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:37191
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225929848825-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40217,127.0.98.61:37191,127.0.98.60:45693
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 05:01:20.604676 13764 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37191 } attempt: 1
I20260924 05:01:20.605201 13766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:20.605035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37191 (local address 127.0.0.1:52442)
0924 05:01:20.605105 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:20.605158 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260924 05:01:20.605561 13766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:20.605474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37191 (local address 127.0.0.1:52450)
0924 05:01:20.605512 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:20.605539 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260924 05:01:20.605655 13764 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:37191: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37191: connect: Connection refused (error 111)
W20260924 05:01:20.654333 13771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:01:20.654547 13771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:01:20.654568 13771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:01:20.655717 13771 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:01:20.655761 13771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:01:20.655773 13771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:01:20.655784 13771 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:01:20.655794 13771 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:01:20.657292 13771 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225929848825-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225929848825-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40217,127.0.98.61:37191,127.0.98.60:45693
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225929848825-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:37191
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225929848825-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13771
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225929848825-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:01:20.657550 13771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:01:20.657749 13771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:01:20.662160 13776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:01:20.662142 13779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:01:20.662123 13777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:01:20.662216 13771 server_base.cc:1034] running on GCE node
I20260924 05:01:20.662572 13771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:01:20.662850 13771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:01:20.663988 13771 hybrid_clock.cc:648] HybridClock initialized: now 1790226080663968 us; error 33 us; skew 500 ppm
Sep 24 05:01:20 dist-test-slave-70st krb5kdc[13683](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226080, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:01:20.666347 13771 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260924 05:01:20.666782 13771 webserver.cc:460] Webserver started at http://127.0.98.61:42977/ using document root <none> and password file <none>
I20260924 05:01:20.666972 13771 fs_manager.cc:359] Metadata directory not provided
I20260924 05:01:20.667019 13771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:01:20.667140 13771 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:01:20.668160 13771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225929848825-392-0/minicluster-data/master-1/data/instance:
uuid: "a1fb6cb9d599402480a132624985096d"
format_stamp: "Formatted at 2026-09-24 05:01:20 on dist-test-slave-70st"
server_key: "b63d5df60f66f434d782cfc764873d33"
server_key_iv: "4a025913fb27a94f805881b066660631"
server_key_version: "encryptionkey@0"
I20260924 05:01:20.668514 13771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225929848825-392-0/minicluster-data/master-1/wal/instance:
uuid: "a1fb6cb9d599402480a132624985096d"
format_stamp: "Formatted at 2026-09-24 05:01:20 on dist-test-slave-70st"
server_key: "b63d5df60f66f434d782cfc764873d33"
server_key_iv: "4a025913fb27a94f805881b066660631"
server_key_version: "encryptionkey@0"
I20260924 05:01:20.669739 13771 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 05:01:20.670406 13786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:01:20.670590 13771 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 05:01:20.670671 13771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225929848825-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225929848825-392-0/minicluster-data/master-1/wal
uuid: "a1fb6cb9d599402480a132624985096d"
format_stamp: "Formatted at 2026-09-24 05:01:20 on dist-test-slave-70st"
server_key: "b63d5df60f66f434d782cfc764873d33"
server_key_iv: "4a025913fb27a94f805881b066660631"
server_key_version: "encryptionkey@0"
I20260924 05:01:20.670732 13771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225929848825-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225929848825-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225929848825-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:01:20.673812 13764 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37191 } attempt: 2
I20260924 05:01:20.674221 13766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:20.674077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37191 (local address 127.0.0.1:52454)
0924 05:01:20.674151 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:20.674190 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260924 05:01:20.674527 13766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:20.674455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37191 (local address 127.0.0.1:52460)
0924 05:01:20.674488 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:20.674510 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260924 05:01:20.674597 13764 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:37191: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37191: connect: Connection refused (error 111)
I20260924 05:01:20.685365 13771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:01:20.687322 13771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:01:20.687433 13771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:01:20.691184 13771 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:37191
I20260924 05:01:20.691234 13838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:37191 every 8 connection(s)
I20260924 05:01:20.691541 13771 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225929848825-392-0/minicluster-data/master-1/data/info.pb
I20260924 05:01:20.692414 13839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:01:20.694012 13839 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40217 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 05:01:20 dist-test-slave-70st krb5kdc[13683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226080, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:01:20.700242 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 13771
I20260924 05:01:20.700346 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225929848825-392-0/minicluster-data/master-1/wal/instance
I20260924 05:01:20.700521 392 external_mini_cluster.cc:1428] Setting key 9c1777dc254cde1efda8e5ed4ead1719
I20260924 05:01:20.702699 13767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:20.694685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58572 (local address 127.0.98.62:40217)
0924 05:01:20.694734 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 05:01:20.694736 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:01:20.694977 (+ 241us) server_negotiation.cc:407] Connection header received
0924 05:01:20.695442 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:01:20.695443 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:01:20.695457 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:01:20.695514 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:01:20.696132 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:20.696646 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:20.697597 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:20.697807 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:20.697863 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:20.699807 (+ 1944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:01:20.699808 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:01:20.699808 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:01:20.699815 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:01:20.701347 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:01:20.701736 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:01:20.701737 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:01:20.701738 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:01:20.701809 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:01:20.702078 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:01:20.702079 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:01:20.702079 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:01:20.702189 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:01:20.702251 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 05:01:20.702562 (+ 311us) server_negotiation.cc:299] Negotiation successful
0924 05:01:20.702633 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 05:01:20.702914 13841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:20.694668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40217 (local address 127.0.0.1:58572)
0924 05:01:20.694826 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:20.694863 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 05:01:20.695322 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:01:20.695578 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:01:20.695581 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:01:20.695835 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:01:20.696039 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:01:20.696044 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:01:20.696711 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:01:20.696712 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:01:20.697531 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:01:20.697533 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:01:20.697867 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:01:20.697867 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:01:20.697970 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:20.698387 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:01:20.698394 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:01:20.699731 (+ 1337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:01:20.701449 (+ 1718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:01:20.701457 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:01:20.701459 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:01:20.701668 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:01:20.701878 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:01:20.701878 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:01:20.701878 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:01:20.702017 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:01:20.702250 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:01:20.702255 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:01:20.702428 (+ 173us) client_negotiation.cc:769] Sending connection context
0924 05:01:20.702537 (+ 109us) client_negotiation.cc:240] Negotiation successful
0924 05:01:20.702694 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":78,"threads_started":1}
I20260924 05:01:20.703444 13839 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37191 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Sep 24 05:01:20 dist-test-slave-70st krb5kdc[13683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226080, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260924 05:01:20.710925 13841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:20.703690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37191 (local address 127.0.0.1:52462)
0924 05:01:20.703738 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:20.703750 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 05:01:20.703791 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:01:20.704062 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:01:20.704063 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:01:20.704200 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:01:20.704261 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:01:20.704263 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:01:20.705128 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:01:20.705129 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:01:20.705991 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:01:20.705994 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:01:20.706484 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:01:20.706485 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:01:20.706558 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:20.706983 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:01:20.706989 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:01:20.708325 (+ 1336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:01:20.709899 (+ 1574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:01:20.709900 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:01:20.709900 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:01:20.710076 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:01:20.710271 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:01:20.710272 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:01:20.710273 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:01:20.710336 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:01:20.710667 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:01:20.710667 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:01:20.710700 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 05:01:20.710759 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 05:01:20.710852 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260924 05:01:20.711143 13843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:20.703751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52462 (local address 127.0.98.61:37191)
0924 05:01:20.703912 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 05:01:20.703917 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 05:01:20.703936 (+ 19us) server_negotiation.cc:407] Connection header received
0924 05:01:20.703960 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:01:20.703962 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:01:20.703974 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:01:20.704044 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:01:20.704354 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:20.705045 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:20.706079 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:20.706408 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:20.706467 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:20.708403 (+ 1936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:01:20.708411 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:01:20.708411 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:01:20.708417 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:01:20.709823 (+ 1406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:01:20.710143 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:01:20.710145 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:01:20.710146 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:01:20.710197 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:01:20.710404 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:01:20.710405 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:01:20.710405 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:01:20.710569 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:01:20.710644 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 05:01:20.710796 (+ 152us) server_negotiation.cc:299] Negotiation successful
0924 05:01:20.710951 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":131,"thread_start_us":91,"threads_started":1}
I20260924 05:01:20.711711 13839 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45693 } has no permanent_uuid. Determining permanent_uuid...
I20260924 05:01:20.712261 13841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:20.712139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45693 (local address 127.0.0.1:39974)
0924 05:01:20.712185 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:20.712238 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260924 05:01:20.712411 13789 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45693: connect: Connection refused (error 111)
I20260924 05:01:20.712796 13841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:20.712710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45693 (local address 127.0.0.1:39986)
0924 05:01:20.712749 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:20.712776 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260924 05:01:20.712893 13839 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45693: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45693: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225929848825-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225929848825-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260924 05:01:20.726003 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225929848825-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225929848825-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225929848825-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225929848825-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:45693
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225929848825-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40217,127.0.98.61:37191,127.0.98.60:45693
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 05:01:20.762130 13839 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45693 } attempt: 1
I20260924 05:01:20.762674 13841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:20.762524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45693 (local address 127.0.0.1:39994)
0924 05:01:20.762584 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:20.762636 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260924 05:01:20.763031 13841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:20.762938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45693 (local address 127.0.0.1:40004)
0924 05:01:20.762980 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:20.763009 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260924 05:01:20.763178 13839 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45693: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45693: connect: Connection refused (error 111)
I20260924 05:01:20.765753 13764 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37191 } attempt: 3
Sep 24 05:01:20 dist-test-slave-70st krb5kdc[13683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226080, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 05:01:20.772692 13843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:20.766169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52478 (local address 127.0.98.61:37191)
0924 05:01:20.766233 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 05:01:20.766235 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:01:20.766250 (+ 15us) server_negotiation.cc:407] Connection header received
0924 05:01:20.766282 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:01:20.766283 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:01:20.766298 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:01:20.766366 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:01:20.766677 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:20.767477 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:20.768156 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:20.768476 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:20.768562 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:20.770459 (+ 1897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:01:20.770461 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:01:20.770461 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:01:20.770467 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:01:20.771692 (+ 1225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:01:20.772070 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:01:20.772070 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:01:20.772071 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:01:20.772115 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:01:20.772289 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:01:20.772289 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:01:20.772289 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:01:20.772382 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:01:20.772424 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0924 05:01:20.772567 (+ 143us) server_negotiation.cc:299] Negotiation successful
0924 05:01:20.772623 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 05:01:20.772805 13766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:20.766094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37191 (local address 127.0.0.1:52478)
0924 05:01:20.766156 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:20.766168 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 05:01:20.766230 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:01:20.766382 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:01:20.766383 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:01:20.766543 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:01:20.766608 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:01:20.766610 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:01:20.767574 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:01:20.767575 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:01:20.768081 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:01:20.768083 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:01:20.768576 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:01:20.768577 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:01:20.768646 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:20.769088 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:01:20.769092 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:01:20.770370 (+ 1278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:01:20.771772 (+ 1402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:01:20.771773 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:01:20.771774 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:01:20.772000 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:01:20.772168 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:01:20.772169 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:01:20.772169 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:01:20.772236 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:01:20.772432 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:01:20.772433 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:01:20.772481 (+ 48us) client_negotiation.cc:769] Sending connection context
0924 05:01:20.772554 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 05:01:20.772649 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260924 05:01:20.773254 13764 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45693 } has no permanent_uuid. Determining permanent_uuid...
I20260924 05:01:20.773629 13766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:20.773541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45693 (local address 127.0.0.1:40008)
0924 05:01:20.773578 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:20.773607 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260924 05:01:20.773945 13766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:20.773866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45693 (local address 127.0.0.1:40022)
0924 05:01:20.773905 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:20.773927 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260924 05:01:20.774025 13764 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45693: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45693: connect: Connection refused (error 111)
I20260924 05:01:20.805193 13764 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45693 } attempt: 1
I20260924 05:01:20.805719 13766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:20.805595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45693 (local address 127.0.0.1:40038)
0924 05:01:20.805648 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:20.805682 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260924 05:01:20.806200 13766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:20.806075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45693 (local address 127.0.0.1:40044)
0924 05:01:20.806144 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:20.806172 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260924 05:01:20.806311 13764 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45693: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45693: connect: Connection refused (error 111)
W20260924 05:01:20.809461 13846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:01:20.809696 13846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:01:20.809731 13846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:01:20.810938 13846 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:01:20.810987 13846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:01:20.811010 13846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:01:20.811029 13846 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:01:20.811050 13846 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:01:20.812685 13846 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225929848825-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225929848825-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40217,127.0.98.61:37191,127.0.98.60:45693
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225929848825-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:45693
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225929848825-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13846
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225929848825-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:01:20.812961 13846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:01:20.813185 13846 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:01:20.817991 13851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:01:20.817991 13854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:01:20.818148 13846 server_base.cc:1034] running on GCE node
W20260924 05:01:20.817994 13852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:01:20.818470 13846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:01:20.818756 13846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:01:20.819916 13846 hybrid_clock.cc:648] HybridClock initialized: now 1790226080819896 us; error 36 us; skew 500 ppm
Sep 24 05:01:20 dist-test-slave-70st krb5kdc[13683](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226080, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:01:20.822638 13846 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260924 05:01:20.823108 13846 webserver.cc:460] Webserver started at http://127.0.98.60:45345/ using document root <none> and password file <none>
I20260924 05:01:20.823315 13846 fs_manager.cc:359] Metadata directory not provided
I20260924 05:01:20.823370 13846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:01:20.823499 13846 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:01:20.824777 13846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225929848825-392-0/minicluster-data/master-2/data/instance:
uuid: "843596b0dde44b45b001a58f39255add"
format_stamp: "Formatted at 2026-09-24 05:01:20 on dist-test-slave-70st"
server_key: "3a76810a8d8554c07ad5a73b2cfa28f4"
server_key_iv: "043b6349d9024e8fdd41a04d70da8f20"
server_key_version: "encryptionkey@0"
I20260924 05:01:20.825122 13846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225929848825-392-0/minicluster-data/master-2/wal/instance:
uuid: "843596b0dde44b45b001a58f39255add"
format_stamp: "Formatted at 2026-09-24 05:01:20 on dist-test-slave-70st"
server_key: "3a76810a8d8554c07ad5a73b2cfa28f4"
server_key_iv: "043b6349d9024e8fdd41a04d70da8f20"
server_key_version: "encryptionkey@0"
I20260924 05:01:20.826436 13846 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 05:01:20.827180 13861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:01:20.827358 13846 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:01:20.827440 13846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225929848825-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225929848825-392-0/minicluster-data/master-2/wal
uuid: "843596b0dde44b45b001a58f39255add"
format_stamp: "Formatted at 2026-09-24 05:01:20 on dist-test-slave-70st"
server_key: "3a76810a8d8554c07ad5a73b2cfa28f4"
server_key_iv: "043b6349d9024e8fdd41a04d70da8f20"
server_key_version: "encryptionkey@0"
I20260924 05:01:20.827507 13846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225929848825-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225929848825-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225929848825-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:01:20.831332 13839 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45693 } attempt: 2
I20260924 05:01:20.831724 13841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:20.831616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45693 (local address 127.0.0.1:40046)
0924 05:01:20.831666 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:20.831696 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260924 05:01:20.832017 13841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:20.831943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45693 (local address 127.0.0.1:40058)
0924 05:01:20.831976 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:20.831999 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260924 05:01:20.832089 13839 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45693: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45693: connect: Connection refused (error 111)
I20260924 05:01:20.842883 13846 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:01:20.844666 13846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:01:20.844805 13846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:01:20.848836 13846 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:45693
I20260924 05:01:20.848881 13913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:45693 every 8 connection(s)
I20260924 05:01:20.849201 13846 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225929848825-392-0/minicluster-data/master-2/data/info.pb
I20260924 05:01:20.850037 13914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:01:20.850705 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 13846
I20260924 05:01:20.850821 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790225929848825-392-0/minicluster-data/master-2/wal/instance
I20260924 05:01:20.850982 392 external_mini_cluster.cc:1428] Setting key 105cab20a7af7eea50ff8d1106d002de
I20260924 05:01:20.851854 13914 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40217 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 05:01:20 dist-test-slave-70st krb5kdc[13683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226080, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 24 05:01:20 dist-test-slave-70st krb5kdc[13683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226080, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 05:01:20.860081 13767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:20.852863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58582 (local address 127.0.98.62:40217)
0924 05:01:20.852915 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 05:01:20.852917 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:01:20.853072 (+ 155us) server_negotiation.cc:407] Connection header received
0924 05:01:20.853133 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:01:20.853134 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:01:20.853144 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:01:20.853190 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:01:20.853696 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:20.854159 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:20.854993 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:20.855196 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:20.855253 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:20.857276 (+ 2023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:01:20.857278 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:01:20.857278 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:01:20.857285 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:01:20.858717 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:01:20.859186 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:01:20.859187 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:01:20.859187 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:01:20.859234 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:01:20.859464 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:01:20.859464 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:01:20.859465 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:01:20.859573 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:01:20.859628 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 05:01:20.859908 (+ 280us) server_negotiation.cc:299] Negotiation successful
0924 05:01:20.859987 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 05:01:20.860239 13918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:20.852804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40217 (local address 127.0.0.1:58582)
0924 05:01:20.852982 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:20.853014 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 05:01:20.853074 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:01:20.853205 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:01:20.853208 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:01:20.853525 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:01:20.853619 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:01:20.853624 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:01:20.854237 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:01:20.854238 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:01:20.854908 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:01:20.854910 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:01:20.855265 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:01:20.855266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:01:20.855329 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:20.855790 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:01:20.855798 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:01:20.857200 (+ 1402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:01:20.858815 (+ 1615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:01:20.858821 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:01:20.858823 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:01:20.859120 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:01:20.859299 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:01:20.859300 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:01:20.859300 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:01:20.859407 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:01:20.859630 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:01:20.859636 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:01:20.859771 (+ 135us) client_negotiation.cc:769] Sending connection context
0924 05:01:20.859870 (+ 99us) client_negotiation.cc:240] Negotiation successful
0924 05:01:20.860021 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":106,"threads_started":1}
I20260924 05:01:20.860251 13917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:20.851483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40060 (local address 127.0.98.60:45693)
0924 05:01:20.851669 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 05:01:20.851680 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 05:01:20.851713 (+ 33us) server_negotiation.cc:407] Connection header received
0924 05:01:20.852183 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:01:20.852192 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:01:20.852261 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:01:20.852414 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:01:20.853162 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:20.854059 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:20.854748 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:20.855079 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:20.855171 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:20.857740 (+ 2569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:01:20.857748 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:01:20.857748 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:01:20.857755 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:01:20.859231 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:01:20.859527 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:01:20.859531 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:01:20.859532 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:01:20.859584 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:01:20.859742 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:01:20.859743 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:01:20.859744 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:01:20.859880 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:01:20.860003 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0924 05:01:20.860079 (+ 76us) server_negotiation.cc:299] Negotiation successful
0924 05:01:20.860149 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":87,"threads_started":1}
I20260924 05:01:20.860984 13914 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37191 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 05:01:20 dist-test-slave-70st krb5kdc[13683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226080, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 24 05:01:20 dist-test-slave-70st krb5kdc[13683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226080, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:01:20.867034 13918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:20.861202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37191 (local address 127.0.0.1:52486)
0924 05:01:20.861231 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:20.861239 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 05:01:20.861268 (+ 29us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:01:20.861431 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:01:20.861431 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:01:20.861556 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:01:20.861588 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:01:20.861590 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:01:20.862177 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:01:20.862178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:01:20.862653 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:01:20.862655 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:01:20.862976 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:01:20.862977 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:01:20.863031 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:20.863376 (+ 345us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:01:20.863380 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:01:20.864478 (+ 1098us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:01:20.866114 (+ 1636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:01:20.866115 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:01:20.866116 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:01:20.866283 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:01:20.866461 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:01:20.866461 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:01:20.866462 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:01:20.866506 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:01:20.866737 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:01:20.866737 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:01:20.866782 (+ 45us) client_negotiation.cc:769] Sending connection context
0924 05:01:20.866857 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 05:01:20.866957 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15}
I20260924 05:01:20.867038 13843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:20.861263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52486 (local address 127.0.98.61:37191)
0924 05:01:20.861313 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 05:01:20.861315 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:01:20.861322 (+ 7us) server_negotiation.cc:407] Connection header received
0924 05:01:20.861345 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:01:20.861346 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:01:20.861355 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:01:20.861404 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:01:20.861650 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:20.862112 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:20.862720 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:20.862923 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:20.862975 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:20.864556 (+ 1581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:01:20.864558 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:01:20.864558 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:01:20.864565 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:01:20.866034 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:01:20.866333 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:01:20.866334 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:01:20.866334 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:01:20.866398 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:01:20.866552 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:01:20.866552 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:01:20.866552 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:01:20.866682 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:01:20.866738 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 05:01:20.866878 (+ 140us) server_negotiation.cc:299] Negotiation successful
0924 05:01:20.866950 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 05:01:20.867413 13914 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45693 } has no permanent_uuid. Determining permanent_uuid...
I20260924 05:01:20.867933 13767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:20.861095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58596 (local address 127.0.98.62:40217)
0924 05:01:20.861160 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 05:01:20.861162 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:01:20.861171 (+ 9us) server_negotiation.cc:407] Connection header received
0924 05:01:20.861191 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:01:20.861192 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:01:20.861201 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:01:20.861253 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:01:20.861773 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:20.862232 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:20.862840 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:20.863036 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:20.863087 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:20.865165 (+ 2078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:01:20.865167 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:01:20.865167 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:01:20.865175 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:01:20.866609 (+ 1434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:01:20.866935 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:01:20.866936 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:01:20.866936 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:01:20.866990 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:01:20.867233 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:01:20.867234 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:01:20.867234 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:01:20.867370 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:01:20.867553 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0924 05:01:20.867761 (+ 208us) server_negotiation.cc:299] Negotiation successful
0924 05:01:20.867845 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
Sep 24 05:01:20 dist-test-slave-70st krb5kdc[13683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226080, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 24 05:01:20 dist-test-slave-70st krb5kdc[13683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226080, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 05:01:20.873449 13918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:20.867647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45693 (local address 127.0.0.1:40076)
0924 05:01:20.867686 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:20.867694 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 05:01:20.867762 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:01:20.867913 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:01:20.867914 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:01:20.868039 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:01:20.868072 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:01:20.868073 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:01:20.868739 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:01:20.868739 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:01:20.869191 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:01:20.869192 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:01:20.869584 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:01:20.869585 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:01:20.869631 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:20.869973 (+ 342us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:01:20.869976 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:01:20.871004 (+ 1028us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:01:20.872513 (+ 1509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:01:20.872513 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:01:20.872514 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:01:20.872679 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:01:20.872829 (+ 150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:01:20.872830 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:01:20.872830 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:01:20.872888 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:01:20.873102 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:01:20.873103 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:01:20.873140 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 05:01:20.873200 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 05:01:20.873294 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260924 05:01:20.873514 13917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:20.867707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40076 (local address 127.0.98.60:45693)
0924 05:01:20.867738 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0924 05:01:20.867740 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:01:20.867745 (+ 5us) server_negotiation.cc:407] Connection header received
0924 05:01:20.867816 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:01:20.867816 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:01:20.867826 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:01:20.867849 (+ 23us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:01:20.868144 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:20.868680 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:20.869256 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:20.869520 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:20.869571 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:20.871074 (+ 1503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:01:20.871075 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:01:20.871076 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:01:20.871082 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:01:20.872453 (+ 1371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:01:20.872729 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:01:20.872730 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:01:20.872730 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:01:20.872779 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:01:20.872937 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:01:20.872938 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:01:20.872938 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:01:20.873043 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:01:20.873095 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 05:01:20.873222 (+ 127us) server_negotiation.cc:299] Negotiation successful
0924 05:01:20.873344 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":61,"server-negotiator.queue_time_us":18}
I20260924 05:01:20.875059 13914 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 843596b0dde44b45b001a58f39255add: Bootstrap starting.
I20260924 05:01:20.875492 13843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:20.868549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52490 (local address 127.0.98.61:37191)
0924 05:01:20.868586 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 05:01:20.868588 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:01:20.868596 (+ 8us) server_negotiation.cc:407] Connection header received
0924 05:01:20.868635 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:01:20.868636 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:01:20.868644 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:01:20.868687 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:01:20.869096 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:20.869608 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:20.870324 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:20.870521 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:20.870573 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:20.872828 (+ 2255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:01:20.872830 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:01:20.872830 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:01:20.872838 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:01:20.874473 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:01:20.874790 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:01:20.874791 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:01:20.874791 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:01:20.874841 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:01:20.875008 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:01:20.875008 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:01:20.875009 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:01:20.875112 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:01:20.875293 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0924 05:01:20.875329 (+ 36us) server_negotiation.cc:299] Negotiation successful
0924 05:01:20.875398 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 05:01:20.875967 13914 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 843596b0dde44b45b001a58f39255add: Neither blocks nor log segments found. Creating new log.
I20260924 05:01:20.876264 13914 log.cc:824] T 00000000000000000000000000000000 P 843596b0dde44b45b001a58f39255add: Log is configured to *not* fsync() on all Append() calls
I20260924 05:01:20.877025 13914 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 843596b0dde44b45b001a58f39255add: No bootstrap required, opened a new log
I20260924 05:01:20.877048 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 05:01:20.878996 13914 raft_consensus.cc:348] T 00000000000000000000000000000000 P 843596b0dde44b45b001a58f39255add [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea6743375da4d068e0f234d2f09636b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40217 } } peers { permanent_uuid: "a1fb6cb9d599402480a132624985096d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37191 } } peers { permanent_uuid: "843596b0dde44b45b001a58f39255add" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45693 } }
I20260924 05:01:20.879163 13914 raft_consensus.cc:374] T 00000000000000000000000000000000 P 843596b0dde44b45b001a58f39255add [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:01:20.879197 13914 raft_consensus.cc:729] T 00000000000000000000000000000000 P 843596b0dde44b45b001a58f39255add [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 843596b0dde44b45b001a58f39255add, State: Initialized, Role: FOLLOWER
I20260924 05:01:20.879344 13914 consensus_queue.cc:260] T 00000000000000000000000000000000 P 843596b0dde44b45b001a58f39255add [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea6743375da4d068e0f234d2f09636b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40217 } } peers { permanent_uuid: "a1fb6cb9d599402480a132624985096d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37191 } } peers { permanent_uuid: "843596b0dde44b45b001a58f39255add" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45693 } }
I20260924 05:01:20.879662 13927 sys_catalog.cc:455] T 00000000000000000000000000000000 P 843596b0dde44b45b001a58f39255add [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea6743375da4d068e0f234d2f09636b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40217 } } peers { permanent_uuid: "a1fb6cb9d599402480a132624985096d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37191 } } peers { permanent_uuid: "843596b0dde44b45b001a58f39255add" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45693 } } }
I20260924 05:01:20.879709 13914 sys_catalog.cc:564] T 00000000000000000000000000000000 P 843596b0dde44b45b001a58f39255add [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:01:20.881117 13927 sys_catalog.cc:458] T 00000000000000000000000000000000 P 843596b0dde44b45b001a58f39255add [sys.catalog]: This master's current role is: FOLLOWER
I20260924 05:01:20.881455 13764 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45693 } attempt: 2
W20260924 05:01:20.883778 13938 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 843596b0dde44b45b001a58f39255add: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 05:01:20.883993 13938 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 05:01:20.885720 13917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:20.878528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40092 (local address 127.0.98.60:45693)
0924 05:01:20.878563 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 05:01:20.878565 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:01:20.878604 (+ 39us) server_negotiation.cc:407] Connection header received
0924 05:01:20.878674 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:01:20.878675 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:01:20.878685 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:01:20.878729 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:01:20.879248 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:20.879969 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:20.880981 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:20.881278 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:20.881314 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:20.882818 (+ 1504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:01:20.882820 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:01:20.882820 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:01:20.882827 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:01:20.884524 (+ 1697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:01:20.884935 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:01:20.884936 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:01:20.884936 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:01:20.884993 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:01:20.885174 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:01:20.885175 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:01:20.885175 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:01:20.885325 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:01:20.885450 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0924 05:01:20.885563 (+ 113us) server_negotiation.cc:299] Negotiation successful
0924 05:01:20.885634 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 05:01:20.882555 13846 master_runner.cc:186] Error getting master registration for 127.0.98.62:40217: OK, instance_id { permanent_uuid: "9ea6743375da4d068e0f234d2f09636b" instance_seqno: 1790226080541391 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 05:01:20.886703 13846 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 05:01:20.886947 13843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:20.878410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52492 (local address 127.0.98.61:37191)
0924 05:01:20.878453 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 05:01:20.878456 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:01:20.878679 (+ 223us) server_negotiation.cc:407] Connection header received
0924 05:01:20.878766 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:01:20.878767 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:01:20.878776 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:01:20.878802 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:01:20.879448 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:20.880112 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:20.880974 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:20.881271 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:20.881327 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:20.883040 (+ 1713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:01:20.883041 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:01:20.883042 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:01:20.883049 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:01:20.884484 (+ 1435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:01:20.886144 (+ 1660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:01:20.886145 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:01:20.886146 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:01:20.886201 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:01:20.886409 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:01:20.886410 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:01:20.886411 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:01:20.886533 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:01:20.886598 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 05:01:20.886788 (+ 190us) server_negotiation.cc:299] Negotiation successful
0924 05:01:20.886856 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 05:01:20.888103 13767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:20.878375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58602 (local address 127.0.98.62:40217)
0924 05:01:20.878469 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0924 05:01:20.878471 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:01:20.879841 (+ 1370us) server_negotiation.cc:407] Connection header received
0924 05:01:20.879932 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:01:20.879934 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:01:20.879945 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:01:20.879975 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:01:20.880693 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:20.881386 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:20.882559 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:20.882854 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:20.882891 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:20.885048 (+ 2157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:01:20.885049 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:01:20.885050 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:01:20.885056 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:01:20.886699 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:01:20.887044 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:01:20.887045 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:01:20.887045 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:01:20.887097 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:01:20.887294 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:01:20.887295 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:01:20.887295 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:01:20.887406 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:01:20.887916 (+ 510us) server_negotiation.cc:1035] Waiting for connection context
0924 05:01:20.887947 (+ 31us) server_negotiation.cc:299] Negotiation successful
0924 05:01:20.888016 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
Sep 24 05:01:20 dist-test-slave-70st krb5kdc[13683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226080, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 05:01:20.891052 13766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:20.881695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45693 (local address 127.0.0.1:40096)
0924 05:01:20.881784 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:20.881793 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:01:20.881826 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:01:20.883165 (+ 1339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:01:20.883166 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:01:20.883334 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:01:20.883371 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:01:20.883373 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:01:20.885541 (+ 2168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:01:20.885542 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:01:20.886272 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:01:20.886274 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:01:20.887115 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:01:20.887116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:01:20.887169 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:20.887524 (+ 355us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:01:20.887529 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:01:20.888679 (+ 1150us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:01:20.890165 (+ 1486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:01:20.890166 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:01:20.890167 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:01:20.890314 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:01:20.890518 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:01:20.890519 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:01:20.890519 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:01:20.890574 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:01:20.890770 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:01:20.890771 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:01:20.890811 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 05:01:20.890869 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 05:01:20.890958 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260924 05:01:20.891078 13939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:20.881739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40096 (local address 127.0.98.60:45693)
0924 05:01:20.881878 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 05:01:20.881880 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:01:20.881886 (+ 6us) server_negotiation.cc:407] Connection header received
0924 05:01:20.883030 (+ 1144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:01:20.883032 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:01:20.883053 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:01:20.883744 (+ 691us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:01:20.883798 (+ 54us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:20.885381 (+ 1583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:20.886775 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:20.887031 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:20.887310 (+ 279us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:20.888764 (+ 1454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:01:20.888765 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:01:20.888766 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:01:20.888772 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:01:20.890105 (+ 1333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:01:20.890370 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:01:20.890397 (+ 27us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:01:20.890397 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:01:20.890465 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:01:20.890622 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:01:20.890623 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:01:20.890623 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:01:20.890720 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:01:20.890771 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 05:01:20.890897 (+ 126us) server_negotiation.cc:299] Negotiation successful
0924 05:01:20.890962 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":71,"threads_started":1}
I20260924 05:01:20.892210 13764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ea6743375da4d068e0f234d2f09636b: Bootstrap starting.
I20260924 05:01:20.892782 13764 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9ea6743375da4d068e0f234d2f09636b: Neither blocks nor log segments found. Creating new log.
I20260924 05:01:20.893033 13764 log.cc:824] T 00000000000000000000000000000000 P 9ea6743375da4d068e0f234d2f09636b: Log is configured to *not* fsync() on all Append() calls
I20260924 05:01:20.893827 13764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ea6743375da4d068e0f234d2f09636b: No bootstrap required, opened a new log
I20260924 05:01:20.895211 13764 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ea6743375da4d068e0f234d2f09636b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea6743375da4d068e0f234d2f09636b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40217 } } peers { permanent_uuid: "a1fb6cb9d599402480a132624985096d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37191 } } peers { permanent_uuid: "843596b0dde44b45b001a58f39255add" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45693 } }
I20260924 05:01:20.895320 13764 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9ea6743375da4d068e0f234d2f09636b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:01:20.895340 13764 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ea6743375da4d068e0f234d2f09636b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ea6743375da4d068e0f234d2f09636b, State: Initialized, Role: FOLLOWER
I20260924 05:01:20.895442 13764 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ea6743375da4d068e0f234d2f09636b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea6743375da4d068e0f234d2f09636b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40217 } } peers { permanent_uuid: "a1fb6cb9d599402480a132624985096d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37191 } } peers { permanent_uuid: "843596b0dde44b45b001a58f39255add" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45693 } }
I20260924 05:01:20.895701 13764 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ea6743375da4d068e0f234d2f09636b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:01:20.895689 13942 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ea6743375da4d068e0f234d2f09636b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea6743375da4d068e0f234d2f09636b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40217 } } peers { permanent_uuid: "a1fb6cb9d599402480a132624985096d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37191 } } peers { permanent_uuid: "843596b0dde44b45b001a58f39255add" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45693 } } }
I20260924 05:01:20.895803 13942 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ea6743375da4d068e0f234d2f09636b [sys.catalog]: This master's current role is: FOLLOWER
W20260924 05:01:20.897369 13953 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9ea6743375da4d068e0f234d2f09636b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 05:01:20.897495 13953 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 05:01:20.898123 13696 master_runner.cc:186] Error getting master registration for 127.0.98.61:37191: OK, instance_id { permanent_uuid: "a1fb6cb9d599402480a132624985096d" instance_seqno: 1790226080690148 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 05:01:20.900926 13696 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 05:01:20.923216 13839 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45693 } attempt: 3
Sep 24 05:01:20 dist-test-slave-70st krb5kdc[13683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226080, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 05:01:20.931736 13841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:20.923602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45693 (local address 127.0.0.1:40104)
0924 05:01:20.923751 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:20.923763 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 05:01:20.923804 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:01:20.924056 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:01:20.924057 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:01:20.924244 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:01:20.924295 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:01:20.924298 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:01:20.925250 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:01:20.925250 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:01:20.926110 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:01:20.926113 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:01:20.926610 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:01:20.926611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:01:20.926674 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:20.927180 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:01:20.927185 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:01:20.928727 (+ 1542us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:01:20.930602 (+ 1875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:01:20.930603 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:01:20.930604 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:01:20.930837 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:01:20.931060 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:01:20.931061 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:01:20.931061 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:01:20.931134 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:01:20.931406 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:01:20.931406 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:01:20.931458 (+ 52us) client_negotiation.cc:769] Sending connection context
0924 05:01:20.931524 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 05:01:20.931624 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127}
I20260924 05:01:20.932255 13939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:20.923872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40104 (local address 127.0.98.60:45693)
0924 05:01:20.923910 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 05:01:20.923912 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:01:20.923918 (+ 6us) server_negotiation.cc:407] Connection header received
0924 05:01:20.923939 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:01:20.923940 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:01:20.923952 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:01:20.924008 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:01:20.924381 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:20.925168 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:20.926188 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:20.926535 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:20.926606 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:20.928816 (+ 2210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:01:20.928818 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:01:20.928818 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:01:20.928826 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:01:20.930526 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:01:20.930925 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:01:20.930927 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:01:20.930927 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:01:20.930988 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:01:20.931198 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:01:20.931199 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:01:20.931199 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:01:20.931339 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:01:20.931422 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 05:01:20.932079 (+ 657us) server_negotiation.cc:299] Negotiation successful
0924 05:01:20.932163 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 05:01:20.934413 13839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1fb6cb9d599402480a132624985096d: Bootstrap starting.
I20260924 05:01:20.935179 13839 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a1fb6cb9d599402480a132624985096d: Neither blocks nor log segments found. Creating new log.
I20260924 05:01:20.935496 13839 log.cc:824] T 00000000000000000000000000000000 P a1fb6cb9d599402480a132624985096d: Log is configured to *not* fsync() on all Append() calls
I20260924 05:01:20.936352 13839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1fb6cb9d599402480a132624985096d: No bootstrap required, opened a new log
I20260924 05:01:20.938333 13839 raft_consensus.cc:348] T 00000000000000000000000000000000 P a1fb6cb9d599402480a132624985096d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea6743375da4d068e0f234d2f09636b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40217 } } peers { permanent_uuid: "a1fb6cb9d599402480a132624985096d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37191 } } peers { permanent_uuid: "843596b0dde44b45b001a58f39255add" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45693 } }
I20260924 05:01:20.938480 13839 raft_consensus.cc:374] T 00000000000000000000000000000000 P a1fb6cb9d599402480a132624985096d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:01:20.938512 13839 raft_consensus.cc:729] T 00000000000000000000000000000000 P a1fb6cb9d599402480a132624985096d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1fb6cb9d599402480a132624985096d, State: Initialized, Role: FOLLOWER
I20260924 05:01:20.938691 13839 consensus_queue.cc:260] T 00000000000000000000000000000000 P a1fb6cb9d599402480a132624985096d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea6743375da4d068e0f234d2f09636b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40217 } } peers { permanent_uuid: "a1fb6cb9d599402480a132624985096d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37191 } } peers { permanent_uuid: "843596b0dde44b45b001a58f39255add" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45693 } }
I20260924 05:01:20.939021 13839 sys_catalog.cc:564] T 00000000000000000000000000000000 P a1fb6cb9d599402480a132624985096d [sys.catalog]: configured and running, proceeding with master startup.
W20260924 05:01:20.940840 13967 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a1fb6cb9d599402480a132624985096d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 05:01:20.940943 13967 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 05:01:20.942958 13771 master_runner.cc:263] No leader master found from master a1fb6cb9d599402480a132624985096d
I20260924 05:01:20.943005 13771 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 05:01:20.938984 13956 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1fb6cb9d599402480a132624985096d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea6743375da4d068e0f234d2f09636b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40217 } } peers { permanent_uuid: "a1fb6cb9d599402480a132624985096d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37191 } } peers { permanent_uuid: "843596b0dde44b45b001a58f39255add" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45693 } } }
I20260924 05:01:20.943171 13956 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1fb6cb9d599402480a132624985096d [sys.catalog]: This master's current role is: FOLLOWER
W20260924 05:01:21.039510 13717 tablet.cc:2369] T 00000000000000000000000000000000 P 9ea6743375da4d068e0f234d2f09636b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 05:01:21.095548 13867 tablet.cc:2369] T 00000000000000000000000000000000 P 843596b0dde44b45b001a58f39255add: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 05:01:21.118249 13956 raft_consensus.cc:482] T 00000000000000000000000000000000 P a1fb6cb9d599402480a132624985096d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 05:01:21.118386 13956 raft_consensus.cc:504] T 00000000000000000000000000000000 P a1fb6cb9d599402480a132624985096d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea6743375da4d068e0f234d2f09636b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40217 } } peers { permanent_uuid: "a1fb6cb9d599402480a132624985096d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37191 } } peers { permanent_uuid: "843596b0dde44b45b001a58f39255add" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45693 } }
I20260924 05:01:21.118615 13956 leader_election.cc:288] T 00000000000000000000000000000000 P a1fb6cb9d599402480a132624985096d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ea6743375da4d068e0f234d2f09636b (127.0.98.62:40217), 843596b0dde44b45b001a58f39255add (127.0.98.60:45693)
I20260924 05:01:21.118929 13738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a1fb6cb9d599402480a132624985096d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ea6743375da4d068e0f234d2f09636b" is_pre_election: true
I20260924 05:01:21.118950 13888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a1fb6cb9d599402480a132624985096d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "843596b0dde44b45b001a58f39255add" is_pre_election: true
I20260924 05:01:21.119127 13888 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 843596b0dde44b45b001a58f39255add [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1fb6cb9d599402480a132624985096d in term 0.
I20260924 05:01:21.119135 13738 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9ea6743375da4d068e0f234d2f09636b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1fb6cb9d599402480a132624985096d in term 0.
I20260924 05:01:21.119365 13790 leader_election.cc:302] T 00000000000000000000000000000000 P a1fb6cb9d599402480a132624985096d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ea6743375da4d068e0f234d2f09636b, a1fb6cb9d599402480a132624985096d; no voters:
I20260924 05:01:21.119565 13956 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a1fb6cb9d599402480a132624985096d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 05:01:21.119626 13956 raft_consensus.cc:482] T 00000000000000000000000000000000 P a1fb6cb9d599402480a132624985096d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 05:01:21.119652 13956 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a1fb6cb9d599402480a132624985096d [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:01:21.120383 13956 raft_consensus.cc:504] T 00000000000000000000000000000000 P a1fb6cb9d599402480a132624985096d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea6743375da4d068e0f234d2f09636b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40217 } } peers { permanent_uuid: "a1fb6cb9d599402480a132624985096d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37191 } } peers { permanent_uuid: "843596b0dde44b45b001a58f39255add" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45693 } }
I20260924 05:01:21.120636 13956 leader_election.cc:288] T 00000000000000000000000000000000 P a1fb6cb9d599402480a132624985096d [CANDIDATE]: Term 1 election: Requested vote from peers 9ea6743375da4d068e0f234d2f09636b (127.0.98.62:40217), 843596b0dde44b45b001a58f39255add (127.0.98.60:45693)
I20260924 05:01:21.120664 13888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a1fb6cb9d599402480a132624985096d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "843596b0dde44b45b001a58f39255add"
I20260924 05:01:21.120743 13888 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 843596b0dde44b45b001a58f39255add [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:01:21.120743 13738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a1fb6cb9d599402480a132624985096d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ea6743375da4d068e0f234d2f09636b"
I20260924 05:01:21.120821 13738 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ea6743375da4d068e0f234d2f09636b [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:01:21.121683 13738 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9ea6743375da4d068e0f234d2f09636b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1fb6cb9d599402480a132624985096d in term 1.
I20260924 05:01:21.121683 13888 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 843596b0dde44b45b001a58f39255add [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1fb6cb9d599402480a132624985096d in term 1.
I20260924 05:01:21.121846 13790 leader_election.cc:302] T 00000000000000000000000000000000 P a1fb6cb9d599402480a132624985096d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ea6743375da4d068e0f234d2f09636b, a1fb6cb9d599402480a132624985096d; no voters:
I20260924 05:01:21.121991 13956 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a1fb6cb9d599402480a132624985096d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:01:21.122170 13956 raft_consensus.cc:686] T 00000000000000000000000000000000 P a1fb6cb9d599402480a132624985096d [term 1 LEADER]: Becoming Leader. State: Replica: a1fb6cb9d599402480a132624985096d, State: Running, Role: LEADER
I20260924 05:01:21.122272 13956 consensus_queue.cc:237] T 00000000000000000000000000000000 P a1fb6cb9d599402480a132624985096d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea6743375da4d068e0f234d2f09636b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40217 } } peers { permanent_uuid: "a1fb6cb9d599402480a132624985096d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37191 } } peers { permanent_uuid: "843596b0dde44b45b001a58f39255add" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45693 } }
I20260924 05:01:21.122694 13969 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1fb6cb9d599402480a132624985096d [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1fb6cb9d599402480a132624985096d. Latest consensus state: current_term: 1 leader_uuid: "a1fb6cb9d599402480a132624985096d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea6743375da4d068e0f234d2f09636b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40217 } } peers { permanent_uuid: "a1fb6cb9d599402480a132624985096d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37191 } } peers { permanent_uuid: "843596b0dde44b45b001a58f39255add" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45693 } } }
I20260924 05:01:21.122767 13969 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1fb6cb9d599402480a132624985096d [sys.catalog]: This master's current role is: LEADER
I20260924 05:01:21.122944 13971 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:01:21.123113 13971 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:01:21.124536 13738 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9ea6743375da4d068e0f234d2f09636b [term 1 FOLLOWER]: Refusing update from remote peer a1fb6cb9d599402480a132624985096d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:01:21.124553 13888 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 843596b0dde44b45b001a58f39255add [term 1 FOLLOWER]: Refusing update from remote peer a1fb6cb9d599402480a132624985096d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:01:21.124841 13969 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a1fb6cb9d599402480a132624985096d [LEADER]: Connected to new peer: Peer: permanent_uuid: "843596b0dde44b45b001a58f39255add" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:01:21.124971 13956 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a1fb6cb9d599402480a132624985096d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ea6743375da4d068e0f234d2f09636b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:01:21.126163 13927 sys_catalog.cc:455] T 00000000000000000000000000000000 P 843596b0dde44b45b001a58f39255add [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1fb6cb9d599402480a132624985096d. Latest consensus state: current_term: 1 leader_uuid: "a1fb6cb9d599402480a132624985096d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea6743375da4d068e0f234d2f09636b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40217 } } peers { permanent_uuid: "a1fb6cb9d599402480a132624985096d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37191 } } peers { permanent_uuid: "843596b0dde44b45b001a58f39255add" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45693 } } }
I20260924 05:01:21.126255 13927 sys_catalog.cc:458] T 00000000000000000000000000000000 P 843596b0dde44b45b001a58f39255add [sys.catalog]: This master's current role is: FOLLOWER
I20260924 05:01:21.126199 13942 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ea6743375da4d068e0f234d2f09636b [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1fb6cb9d599402480a132624985096d. Latest consensus state: current_term: 1 leader_uuid: "a1fb6cb9d599402480a132624985096d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea6743375da4d068e0f234d2f09636b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40217 } } peers { permanent_uuid: "a1fb6cb9d599402480a132624985096d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37191 } } peers { permanent_uuid: "843596b0dde44b45b001a58f39255add" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45693 } } }
I20260924 05:01:21.126295 13942 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ea6743375da4d068e0f234d2f09636b [sys.catalog]: This master's current role is: FOLLOWER
I20260924 05:01:21.127007 13956 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1fb6cb9d599402480a132624985096d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a1fb6cb9d599402480a132624985096d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea6743375da4d068e0f234d2f09636b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40217 } } peers { permanent_uuid: "a1fb6cb9d599402480a132624985096d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37191 } } peers { permanent_uuid: "843596b0dde44b45b001a58f39255add" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45693 } } }
I20260924 05:01:21.127099 13956 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1fb6cb9d599402480a132624985096d [sys.catalog]: This master's current role is: LEADER
I20260924 05:01:21.127017 13977 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1fb6cb9d599402480a132624985096d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a1fb6cb9d599402480a132624985096d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea6743375da4d068e0f234d2f09636b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40217 } } peers { permanent_uuid: "a1fb6cb9d599402480a132624985096d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37191 } } peers { permanent_uuid: "843596b0dde44b45b001a58f39255add" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45693 } } }
I20260924 05:01:21.127130 13977 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1fb6cb9d599402480a132624985096d [sys.catalog]: This master's current role is: LEADER
I20260924 05:01:21.127154 13972 mvcc.cc:205] Tried to move back new op lower bound from 7332766028283793408 to 7332766028277407744. Current Snapshot: MvccSnapshot[applied={T|T < 7332766028283793408}]
I20260924 05:01:21.127218 13974 mvcc.cc:205] Tried to move back new op lower bound from 7332766028283793408 to 7332766028277407744. Current Snapshot: MvccSnapshot[applied={T|T < 7332766028283793408}]
I20260924 05:01:21.127200 13927 sys_catalog.cc:455] T 00000000000000000000000000000000 P 843596b0dde44b45b001a58f39255add [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a1fb6cb9d599402480a132624985096d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea6743375da4d068e0f234d2f09636b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40217 } } peers { permanent_uuid: "a1fb6cb9d599402480a132624985096d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37191 } } peers { permanent_uuid: "843596b0dde44b45b001a58f39255add" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45693 } } }
I20260924 05:01:21.127286 13927 sys_catalog.cc:458] T 00000000000000000000000000000000 P 843596b0dde44b45b001a58f39255add [sys.catalog]: This master's current role is: FOLLOWER
I20260924 05:01:21.127449 13942 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ea6743375da4d068e0f234d2f09636b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a1fb6cb9d599402480a132624985096d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea6743375da4d068e0f234d2f09636b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40217 } } peers { permanent_uuid: "a1fb6cb9d599402480a132624985096d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37191 } } peers { permanent_uuid: "843596b0dde44b45b001a58f39255add" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45693 } } }
I20260924 05:01:21.127527 13942 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ea6743375da4d068e0f234d2f09636b [sys.catalog]: This master's current role is: FOLLOWER
I20260924 05:01:21.127848 13971 catalog_manager.cc:1347] Generated new cluster ID: d6e1eb2b0f474af0b6b6f341df2d48fe
I20260924 05:01:21.127898 13971 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:01:21.147590 13971 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:01:21.148252 13971 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:01:21.155676 13971 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a1fb6cb9d599402480a132624985096d: Generated new TSK 0
I20260924 05:01:21.155910 13971 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 05:01:21.884496 13938 catalog_manager.cc:1259] Loaded cluster ID: d6e1eb2b0f474af0b6b6f341df2d48fe
I20260924 05:01:21.884580 13938 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 843596b0dde44b45b001a58f39255add: loading cluster ID for follower catalog manager: success
I20260924 05:01:21.885443 13938 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 843596b0dde44b45b001a58f39255add: acquiring CA information for follower catalog manager: success
I20260924 05:01:21.885732 13938 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 843596b0dde44b45b001a58f39255add: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 05:01:21.897957 13953 catalog_manager.cc:1259] Loaded cluster ID: d6e1eb2b0f474af0b6b6f341df2d48fe
I20260924 05:01:21.898034 13953 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9ea6743375da4d068e0f234d2f09636b: loading cluster ID for follower catalog manager: success
I20260924 05:01:21.898859 13953 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9ea6743375da4d068e0f234d2f09636b: acquiring CA information for follower catalog manager: success
I20260924 05:01:21.899154 13953 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9ea6743375da4d068e0f234d2f09636b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 05:01:22.212579 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 13696
I20260924 05:01:22.217149 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 13771
I20260924 05:01:22.221603 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 13846
2026-09-24T05:01:22Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2874 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790225929848825-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:01:22 dist-test-slave-70st krb5kdc[13986](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:01:22 dist-test-slave-70st krb5kdc[13986](info): set up 2 sockets
Sep 24 05:01:22 dist-test-slave-70st krb5kdc[13986](info): commencing operation
krb5kdc: starting...
W20260924 05:01:23.169312 392 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790225929848825-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790225929848825-392-0/krb5kdc/test-user.keytab.
Sep 24 05:01:23 dist-test-slave-70st krb5kdc[13986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226083, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:01:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:01:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 05:01:23.280408 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790225929848825-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790225929848825-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790225929848825-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39213
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39213
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:01:23.360080 13999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:01:23.360306 13999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:01:23.360339 13999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:01:23.361506 13999 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:01:23.361552 13999 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:01:23.361569 13999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:01:23.361584 13999 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:01:23.361598 13999 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:01:23.363029 13999 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790225929848825-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790225929848825-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39213
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39213
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13999
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790225929848825-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:01:23.363267 13999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:01:23.363467 13999 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:01:23.368106 14004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:01:23.368258 13999 server_base.cc:1034] running on GCE node
W20260924 05:01:23.368126 14005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:01:23.368110 14007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:01:23.368608 13999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:01:23.368896 13999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:01:23.370059 13999 hybrid_clock.cc:648] HybridClock initialized: now 1790226083370043 us; error 28 us; skew 500 ppm
Sep 24 05:01:23 dist-test-slave-70st krb5kdc[13986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226083, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:01:23.372429 13999 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 05:01:23.372866 13999 webserver.cc:460] Webserver started at http://127.0.98.62:39055/ using document root <none> and password file <none>
I20260924 05:01:23.373065 13999 fs_manager.cc:359] Metadata directory not provided
I20260924 05:01:23.373112 13999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:01:23.373232 13999 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:01:23.374315 13999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790225929848825-392-0/minicluster-data/master-0/data/instance:
uuid: "8d30f8b6eee5467c8f9ab0fb580e3a44"
format_stamp: "Formatted at 2026-09-24 05:01:23 on dist-test-slave-70st"
server_key: "3bb87ca96044136e0fa0b53fad76e2cb"
server_key_iv: "d1c7a89b5f9b5f6e52486456d771ccf6"
server_key_version: "encryptionkey@0"
I20260924 05:01:23.374641 13999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790225929848825-392-0/minicluster-data/master-0/wal/instance:
uuid: "8d30f8b6eee5467c8f9ab0fb580e3a44"
format_stamp: "Formatted at 2026-09-24 05:01:23 on dist-test-slave-70st"
server_key: "3bb87ca96044136e0fa0b53fad76e2cb"
server_key_iv: "d1c7a89b5f9b5f6e52486456d771ccf6"
server_key_version: "encryptionkey@0"
I20260924 05:01:23.375864 13999 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 05:01:23.376523 14014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:01:23.376681 13999 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 05:01:23.376766 13999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790225929848825-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790225929848825-392-0/minicluster-data/master-0/wal
uuid: "8d30f8b6eee5467c8f9ab0fb580e3a44"
format_stamp: "Formatted at 2026-09-24 05:01:23 on dist-test-slave-70st"
server_key: "3bb87ca96044136e0fa0b53fad76e2cb"
server_key_iv: "d1c7a89b5f9b5f6e52486456d771ccf6"
server_key_version: "encryptionkey@0"
I20260924 05:01:23.376842 13999 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790225929848825-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790225929848825-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790225929848825-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:01:23.386173 13999 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:01:23.387892 13999 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:01:23.388010 13999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:01:23.391634 13999 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39213
I20260924 05:01:23.391662 14066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39213 every 8 connection(s)
I20260924 05:01:23.391974 13999 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:01:23.392722 14067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:01:23.394681 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 13999
I20260924 05:01:23.394767 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790225929848825-392-0/minicluster-data/master-0/wal/instance
I20260924 05:01:23.394963 392 external_mini_cluster.cc:1428] Setting key 119256834a6e3944258a9f15875cc8e1
I20260924 05:01:23.395040 14067 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d30f8b6eee5467c8f9ab0fb580e3a44: Bootstrap starting.
I20260924 05:01:23.395689 14067 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d30f8b6eee5467c8f9ab0fb580e3a44: Neither blocks nor log segments found. Creating new log.
I20260924 05:01:23.395965 14067 log.cc:824] T 00000000000000000000000000000000 P 8d30f8b6eee5467c8f9ab0fb580e3a44: Log is configured to *not* fsync() on all Append() calls
I20260924 05:01:23.396670 14067 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d30f8b6eee5467c8f9ab0fb580e3a44: No bootstrap required, opened a new log
I20260924 05:01:23.398667 14067 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d30f8b6eee5467c8f9ab0fb580e3a44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d30f8b6eee5467c8f9ab0fb580e3a44" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39213 } }
I20260924 05:01:23.398840 14067 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d30f8b6eee5467c8f9ab0fb580e3a44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:01:23.398874 14067 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d30f8b6eee5467c8f9ab0fb580e3a44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d30f8b6eee5467c8f9ab0fb580e3a44, State: Initialized, Role: FOLLOWER
I20260924 05:01:23.399022 14067 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d30f8b6eee5467c8f9ab0fb580e3a44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d30f8b6eee5467c8f9ab0fb580e3a44" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39213 } }
I20260924 05:01:23.399089 14067 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d30f8b6eee5467c8f9ab0fb580e3a44 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:01:23.399118 14067 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d30f8b6eee5467c8f9ab0fb580e3a44 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:01:23.399163 14067 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d30f8b6eee5467c8f9ab0fb580e3a44 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:01:23.399751 14067 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d30f8b6eee5467c8f9ab0fb580e3a44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d30f8b6eee5467c8f9ab0fb580e3a44" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39213 } }
I20260924 05:01:23.399863 14067 leader_election.cc:302] T 00000000000000000000000000000000 P 8d30f8b6eee5467c8f9ab0fb580e3a44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d30f8b6eee5467c8f9ab0fb580e3a44; no voters:
I20260924 05:01:23.400012 14067 leader_election.cc:288] T 00000000000000000000000000000000 P 8d30f8b6eee5467c8f9ab0fb580e3a44 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:01:23.400067 14072 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d30f8b6eee5467c8f9ab0fb580e3a44 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:01:23.400234 14072 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d30f8b6eee5467c8f9ab0fb580e3a44 [term 1 LEADER]: Becoming Leader. State: Replica: 8d30f8b6eee5467c8f9ab0fb580e3a44, State: Running, Role: LEADER
I20260924 05:01:23.400254 14067 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d30f8b6eee5467c8f9ab0fb580e3a44 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:01:23.400384 14072 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d30f8b6eee5467c8f9ab0fb580e3a44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d30f8b6eee5467c8f9ab0fb580e3a44" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39213 } }
I20260924 05:01:23.400724 14073 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d30f8b6eee5467c8f9ab0fb580e3a44 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8d30f8b6eee5467c8f9ab0fb580e3a44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d30f8b6eee5467c8f9ab0fb580e3a44" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39213 } } }
I20260924 05:01:23.400802 14073 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d30f8b6eee5467c8f9ab0fb580e3a44 [sys.catalog]: This master's current role is: LEADER
I20260924 05:01:23.400998 14074 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d30f8b6eee5467c8f9ab0fb580e3a44 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d30f8b6eee5467c8f9ab0fb580e3a44. Latest consensus state: current_term: 1 leader_uuid: "8d30f8b6eee5467c8f9ab0fb580e3a44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d30f8b6eee5467c8f9ab0fb580e3a44" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39213 } } }
I20260924 05:01:23.401350 14078 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 24 05:01:23 dist-test-slave-70st krb5kdc[13986](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226083, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:01:23.402000 14078 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:01:23.402594 14074 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d30f8b6eee5467c8f9ab0fb580e3a44 [sys.catalog]: This master's current role is: LEADER
I20260924 05:01:23.403487 14078 catalog_manager.cc:1347] Generated new cluster ID: 8a2ba12773dc4cbfb46490b4698e5135
I20260924 05:01:23.403537 14078 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:01:23.405817 14070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:23.395413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55182 (local address 127.0.98.62:39213)
0924 05:01:23.395584 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 05:01:23.395593 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 05:01:23.395619 (+ 26us) server_negotiation.cc:407] Connection header received
0924 05:01:23.396055 (+ 436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:01:23.396064 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:01:23.396123 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:01:23.396283 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:01:23.396971 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:23.397907 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:23.398931 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:23.399443 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:23.399523 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:23.402197 (+ 2674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:01:23.402201 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:01:23.402201 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:01:23.402213 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:01:23.404291 (+ 2078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:01:23.404664 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:01:23.404666 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:01:23.404667 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:01:23.404755 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:01:23.404945 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:01:23.404945 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:01:23.404946 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:01:23.405208 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:01:23.405345 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0924 05:01:23.405491 (+ 146us) server_negotiation.cc:299] Negotiation successful
0924 05:01:23.405618 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":79,"threads_started":1}
I20260924 05:01:23.406762 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 05:01:23.423719 14078 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:01:23.424250 14078 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:01:23.428273 14078 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d30f8b6eee5467c8f9ab0fb580e3a44: Generated new TSK 0
I20260924 05:01:23.428454 14078 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 05:01:23.460757 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 13999
2026-09-24T05:01:23Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1237 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:01:23 dist-test-slave-70st krb5kdc[14094](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:01:23 dist-test-slave-70st krb5kdc[14094](info): set up 2 sockets
Sep 24 05:01:23 dist-test-slave-70st krb5kdc[14094](info): commencing operation
krb5kdc: starting...
W20260924 05:01:24.405246 392 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/krb5kdc/test-user.keytab.
Sep 24 05:01:24 dist-test-slave-70st krb5kdc[14094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226084, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:01:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:01:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 05:01:24.516682 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43939
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43939
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:01:24.594847 14107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:01:24.595077 14107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:01:24.595106 14107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:01:24.596239 14107 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:01:24.596278 14107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:01:24.596304 14107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:01:24.596319 14107 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:01:24.596335 14107 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:01:24.597874 14107 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43939
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43939
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14107
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:01:24.598106 14107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:01:24.598320 14107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:01:24.602874 14115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:01:24.602923 14107 server_base.cc:1034] running on GCE node
W20260924 05:01:24.602874 14113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:01:24.602887 14112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:01:24.603338 14107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:01:24.603619 14107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:01:24.604773 14107 hybrid_clock.cc:648] HybridClock initialized: now 1790226084604758 us; error 28 us; skew 500 ppm
Sep 24 05:01:24 dist-test-slave-70st krb5kdc[14094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226084, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:01:24.607154 14107 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 05:01:24.607575 14107 webserver.cc:460] Webserver started at http://127.0.98.62:34763/ using document root <none> and password file <none>
I20260924 05:01:24.607779 14107 fs_manager.cc:359] Metadata directory not provided
I20260924 05:01:24.607832 14107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:01:24.607951 14107 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:01:24.608948 14107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/master-0/data/instance:
uuid: "7725fcfb576b446cafbc9944edacc6b4"
format_stamp: "Formatted at 2026-09-24 05:01:24 on dist-test-slave-70st"
server_key: "d37b7160a44d8360610ee8ce5d87c5d9"
server_key_iv: "34c8bfdc26c228542a257500188c6491"
server_key_version: "encryptionkey@0"
I20260924 05:01:24.609266 14107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/master-0/wal/instance:
uuid: "7725fcfb576b446cafbc9944edacc6b4"
format_stamp: "Formatted at 2026-09-24 05:01:24 on dist-test-slave-70st"
server_key: "d37b7160a44d8360610ee8ce5d87c5d9"
server_key_iv: "34c8bfdc26c228542a257500188c6491"
server_key_version: "encryptionkey@0"
I20260924 05:01:24.610558 14107 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 05:01:24.611312 14122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:01:24.611507 14107 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 05:01:24.611596 14107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/master-0/wal
uuid: "7725fcfb576b446cafbc9944edacc6b4"
format_stamp: "Formatted at 2026-09-24 05:01:24 on dist-test-slave-70st"
server_key: "d37b7160a44d8360610ee8ce5d87c5d9"
server_key_iv: "34c8bfdc26c228542a257500188c6491"
server_key_version: "encryptionkey@0"
I20260924 05:01:24.611663 14107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:01:24.623534 14107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:01:24.625583 14107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:01:24.625712 14107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:01:24.629213 14107 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43939
I20260924 05:01:24.629294 14174 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43939 every 8 connection(s)
I20260924 05:01:24.629594 14107 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:01:24.630527 14175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:01:24.630790 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 14107
I20260924 05:01:24.630899 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/master-0/wal/instance
I20260924 05:01:24.631071 392 external_mini_cluster.cc:1428] Setting key f9515b4a8e67a94a4b24c2e477adeff3
I20260924 05:01:24.633255 14175 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7725fcfb576b446cafbc9944edacc6b4: Bootstrap starting.
I20260924 05:01:24.633854 14175 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7725fcfb576b446cafbc9944edacc6b4: Neither blocks nor log segments found. Creating new log.
I20260924 05:01:24.634122 14175 log.cc:824] T 00000000000000000000000000000000 P 7725fcfb576b446cafbc9944edacc6b4: Log is configured to *not* fsync() on all Append() calls
I20260924 05:01:24.634858 14175 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7725fcfb576b446cafbc9944edacc6b4: No bootstrap required, opened a new log
I20260924 05:01:24.637077 14175 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7725fcfb576b446cafbc9944edacc6b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7725fcfb576b446cafbc9944edacc6b4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43939 } }
I20260924 05:01:24.637212 14175 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7725fcfb576b446cafbc9944edacc6b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:01:24.637233 14175 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7725fcfb576b446cafbc9944edacc6b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7725fcfb576b446cafbc9944edacc6b4, State: Initialized, Role: FOLLOWER
I20260924 05:01:24.637352 14175 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7725fcfb576b446cafbc9944edacc6b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7725fcfb576b446cafbc9944edacc6b4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43939 } }
I20260924 05:01:24.637439 14175 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7725fcfb576b446cafbc9944edacc6b4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:01:24.637470 14175 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7725fcfb576b446cafbc9944edacc6b4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:01:24.637516 14175 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7725fcfb576b446cafbc9944edacc6b4 [term 0 FOLLOWER]: Advancing to term 1
Sep 24 05:01:24 dist-test-slave-70st krb5kdc[14094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226084, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:01:24.638087 14175 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7725fcfb576b446cafbc9944edacc6b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7725fcfb576b446cafbc9944edacc6b4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43939 } }
I20260924 05:01:24.638195 14175 leader_election.cc:302] T 00000000000000000000000000000000 P 7725fcfb576b446cafbc9944edacc6b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7725fcfb576b446cafbc9944edacc6b4; no voters:
I20260924 05:01:24.638355 14175 leader_election.cc:288] T 00000000000000000000000000000000 P 7725fcfb576b446cafbc9944edacc6b4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:01:24.638453 14180 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7725fcfb576b446cafbc9944edacc6b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:01:24.638610 14180 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7725fcfb576b446cafbc9944edacc6b4 [term 1 LEADER]: Becoming Leader. State: Replica: 7725fcfb576b446cafbc9944edacc6b4, State: Running, Role: LEADER
I20260924 05:01:24.638684 14175 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7725fcfb576b446cafbc9944edacc6b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:01:24.638756 14180 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7725fcfb576b446cafbc9944edacc6b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7725fcfb576b446cafbc9944edacc6b4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43939 } }
I20260924 05:01:24.639123 14181 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7725fcfb576b446cafbc9944edacc6b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7725fcfb576b446cafbc9944edacc6b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7725fcfb576b446cafbc9944edacc6b4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43939 } } }
I20260924 05:01:24.639338 14181 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7725fcfb576b446cafbc9944edacc6b4 [sys.catalog]: This master's current role is: LEADER
I20260924 05:01:24.639276 14182 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7725fcfb576b446cafbc9944edacc6b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7725fcfb576b446cafbc9944edacc6b4. Latest consensus state: current_term: 1 leader_uuid: "7725fcfb576b446cafbc9944edacc6b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7725fcfb576b446cafbc9944edacc6b4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43939 } } }
I20260924 05:01:24.639562 14182 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7725fcfb576b446cafbc9944edacc6b4 [sys.catalog]: This master's current role is: LEADER
W20260924 05:01:24.640317 14196 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7725fcfb576b446cafbc9944edacc6b4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 05:01:24.641269 14196 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 05:01:24.641340 14189 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:01:24.641520 14189 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:01:24.643784 14178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:24.631540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38662 (local address 127.0.98.62:43939)
0924 05:01:24.631695 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 05:01:24.631704 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 05:01:24.631728 (+ 24us) server_negotiation.cc:407] Connection header received
0924 05:01:24.632132 (+ 404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:01:24.632138 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:01:24.632183 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:01:24.632295 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:01:24.633037 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:24.634016 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:24.635006 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:24.635431 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:24.635518 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:24.638025 (+ 2507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:01:24.638030 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:01:24.638030 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:01:24.638043 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:01:24.642319 (+ 4276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:01:24.642731 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:01:24.642735 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:01:24.642735 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:01:24.642827 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:01:24.643030 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:01:24.643030 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:01:24.643030 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:01:24.643245 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:01:24.643343 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 05:01:24.643484 (+ 141us) server_negotiation.cc:299] Negotiation successful
0924 05:01:24.643585 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":58,"threads_started":1}
I20260924 05:01:24.645799 14189 catalog_manager.cc:1347] Generated new cluster ID: 46f0a1bdd2da4de58d54f3cb3ba37607
I20260924 05:01:24.645853 14189 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:01:24.656783 14189 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:01:24.657312 14189 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:01:24.660043 14189 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7725fcfb576b446cafbc9944edacc6b4: Generated new TSK 0
I20260924 05:01:24.660205 14189 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 05:01:24.684742 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43939
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:01:24.764696 14201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:01:24.764904 14201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:01:24.764936 14201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:01:24.766088 14201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:01:24.766134 14201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:01:24.766158 14201 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 05:01:24.767596 14201 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43939
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14201
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:01:24.767824 14201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:01:24.768054 14201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:01:24.772804 14206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:01:24.772843 14207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:01:24.772902 14209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:01:24.773105 14201 server_base.cc:1034] running on GCE node
I20260924 05:01:24.773275 14201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:01:24.773515 14201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:01:24.774654 14201 hybrid_clock.cc:648] HybridClock initialized: now 1790226084774638 us; error 36 us; skew 500 ppm
Sep 24 05:01:24 dist-test-slave-70st krb5kdc[14094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226084, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:01:24.777037 14201 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 05:01:24.777469 14201 webserver.cc:460] Webserver started at http://127.0.98.1:39289/ using document root <none> and password file <none>
I20260924 05:01:24.777694 14201 fs_manager.cc:359] Metadata directory not provided
I20260924 05:01:24.777736 14201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:01:24.777824 14201 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:01:24.778885 14201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-0/data/instance:
uuid: "c60634e3d40f4fdb9b890b2d27481143"
format_stamp: "Formatted at 2026-09-24 05:01:24 on dist-test-slave-70st"
server_key: "57b7b43a095930ff567c4befa65feeb5"
server_key_iv: "049cc7180b3f42ed314190f68ff24118"
server_key_version: "encryptionkey@0"
I20260924 05:01:24.779177 14201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c60634e3d40f4fdb9b890b2d27481143"
format_stamp: "Formatted at 2026-09-24 05:01:24 on dist-test-slave-70st"
server_key: "57b7b43a095930ff567c4befa65feeb5"
server_key_iv: "049cc7180b3f42ed314190f68ff24118"
server_key_version: "encryptionkey@0"
I20260924 05:01:24.780313 14201 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 05:01:24.780985 14216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:01:24.781152 14201 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 05:01:24.781229 14201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-0/wal
uuid: "c60634e3d40f4fdb9b890b2d27481143"
format_stamp: "Formatted at 2026-09-24 05:01:24 on dist-test-slave-70st"
server_key: "57b7b43a095930ff567c4befa65feeb5"
server_key_iv: "049cc7180b3f42ed314190f68ff24118"
server_key_version: "encryptionkey@0"
I20260924 05:01:24.781291 14201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:01:24.788276 14201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:01:24.790352 14201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:01:24.790483 14201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:01:24.790675 14201 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:01:24.790958 14201 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:01:24.790990 14201 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:01:24.791018 14201 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:01:24.791041 14201 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:01:24.796350 14201 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38433
I20260924 05:01:24.796396 14329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38433 every 8 connection(s)
I20260924 05:01:24.796680 14201 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-0/data/info.pb
I20260924 05:01:24.799194 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 14201
I20260924 05:01:24.799274 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-0/wal/instance
I20260924 05:01:24.799479 392 external_mini_cluster.cc:1428] Setting key 7d9d9e1023731ad57c5661c58c75c49f
Sep 24 05:01:24 dist-test-slave-70st krb5kdc[14094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226084, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:01:24.806000 14178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:24.797282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51145 (local address 127.0.98.62:43939)
0924 05:01:24.797342 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 05:01:24.797344 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:01:24.797572 (+ 228us) server_negotiation.cc:407] Connection header received
0924 05:01:24.798034 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:01:24.798035 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:01:24.798046 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:01:24.798106 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:01:24.798951 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:24.799520 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:24.800834 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:24.801051 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:24.801106 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:24.803457 (+ 2351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:01:24.803458 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:01:24.803458 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:01:24.803466 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:01:24.804679 (+ 1213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:01:24.805073 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:01:24.805074 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:01:24.805074 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:01:24.805127 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:01:24.805339 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:01:24.805340 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:01:24.805341 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:01:24.805492 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:01:24.805549 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 05:01:24.805805 (+ 256us) server_negotiation.cc:299] Negotiation successful
0924 05:01:24.805901 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 05:01:24.806171 14332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:24.797258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43939 (local address 127.0.98.1:51145)
0924 05:01:24.797435 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:24.797475 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 05:01:24.797955 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:01:24.798159 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:01:24.798162 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:01:24.798697 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:01:24.798885 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:01:24.798889 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:01:24.799599 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:01:24.799600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:01:24.800746 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:01:24.800749 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:01:24.801124 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:01:24.801125 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:01:24.801263 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:24.801868 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:01:24.801882 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:01:24.803371 (+ 1489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:01:24.804752 (+ 1381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:01:24.804760 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:01:24.804762 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:01:24.805005 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:01:24.805180 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:01:24.805181 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:01:24.805181 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:01:24.805284 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:01:24.805546 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:01:24.805552 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:01:24.805688 (+ 136us) client_negotiation.cc:769] Sending connection context
0924 05:01:24.805778 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 05:01:24.805941 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":75,"threads_started":1}
I20260924 05:01:24.806763 14330 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43939
I20260924 05:01:24.806874 14330 heartbeater.cc:464] Registering TS with master...
I20260924 05:01:24.807045 14330 heartbeater.cc:511] Master 127.0.98.62:43939 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 05:01:24.807565 14139 ts_manager.cc:194] Registered new tserver with Master: c60634e3d40f4fdb9b890b2d27481143 (127.0.98.1:38433)
I20260924 05:01:24.808169 14139 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:51145
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 05:01:24.823604 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43939
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:01:24.901944 14335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:01:24.902151 14335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:01:24.902174 14335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:01:24.903244 14335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:01:24.903276 14335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:01:24.903301 14335 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 05:01:24.904660 14335 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43939
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14335
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:01:24.904840 14335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:01:24.905012 14335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:01:24.909686 14343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:01:24.909699 14340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:01:24.909864 14335 server_base.cc:1034] running on GCE node
W20260924 05:01:24.909739 14341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:01:24.910315 14335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:01:24.910575 14335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:01:24.911747 14335 hybrid_clock.cc:648] HybridClock initialized: now 1790226084911732 us; error 24 us; skew 500 ppm
Sep 24 05:01:24 dist-test-slave-70st krb5kdc[14094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226084, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:01:24.914556 14335 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 05:01:24.915026 14335 webserver.cc:460] Webserver started at http://127.0.98.2:46749/ using document root <none> and password file <none>
I20260924 05:01:24.915227 14335 fs_manager.cc:359] Metadata directory not provided
I20260924 05:01:24.915283 14335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:01:24.915413 14335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:01:24.916469 14335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-1/data/instance:
uuid: "fdfbafe3ba214448b087cf163e74c75a"
format_stamp: "Formatted at 2026-09-24 05:01:24 on dist-test-slave-70st"
server_key: "d8df35ec213e6c68fead8d0c0c14d84c"
server_key_iv: "781a03fd7a040819393296fa0b07da22"
server_key_version: "encryptionkey@0"
I20260924 05:01:24.916879 14335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-1/wal/instance:
uuid: "fdfbafe3ba214448b087cf163e74c75a"
format_stamp: "Formatted at 2026-09-24 05:01:24 on dist-test-slave-70st"
server_key: "d8df35ec213e6c68fead8d0c0c14d84c"
server_key_iv: "781a03fd7a040819393296fa0b07da22"
server_key_version: "encryptionkey@0"
I20260924 05:01:24.918237 14335 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 05:01:24.918903 14350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:01:24.919059 14335 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:01:24.919137 14335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-1/wal
uuid: "fdfbafe3ba214448b087cf163e74c75a"
format_stamp: "Formatted at 2026-09-24 05:01:24 on dist-test-slave-70st"
server_key: "d8df35ec213e6c68fead8d0c0c14d84c"
server_key_iv: "781a03fd7a040819393296fa0b07da22"
server_key_version: "encryptionkey@0"
I20260924 05:01:24.919194 14335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:01:24.937796 14335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:01:24.939988 14335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:01:24.940125 14335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:01:24.940344 14335 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:01:24.940747 14335 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:01:24.940793 14335 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:01:24.940828 14335 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:01:24.940846 14335 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:01:24.946141 14335 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37425
I20260924 05:01:24.946179 14463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37425 every 8 connection(s)
I20260924 05:01:24.946486 14335 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-1/data/info.pb
I20260924 05:01:24.948058 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 14335
I20260924 05:01:24.948172 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-1/wal/instance
I20260924 05:01:24.948372 392 external_mini_cluster.cc:1428] Setting key f2f51fc60b144642d487a726263ef266
Sep 24 05:01:24 dist-test-slave-70st krb5kdc[14094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226084, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 05:01:24.956403 14178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:24.947188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50385 (local address 127.0.98.62:43939)
0924 05:01:24.947253 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 05:01:24.947256 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:01:24.947456 (+ 200us) server_negotiation.cc:407] Connection header received
0924 05:01:24.947940 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:01:24.947941 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:01:24.947953 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:01:24.948020 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:01:24.949081 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:24.949770 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:24.951143 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:24.951333 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:24.951412 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:24.953712 (+ 2300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:01:24.953713 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:01:24.953714 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:01:24.953721 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:01:24.955103 (+ 1382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:01:24.955509 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:01:24.955510 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:01:24.955510 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:01:24.955570 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:01:24.955780 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:01:24.955781 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:01:24.955781 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:01:24.955917 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:01:24.955973 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 05:01:24.956229 (+ 256us) server_negotiation.cc:299] Negotiation successful
0924 05:01:24.956307 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 05:01:24.956579 14466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:24.947157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43939 (local address 127.0.98.2:50385)
0924 05:01:24.947307 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:24.947371 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0924 05:01:24.947840 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:01:24.948082 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:01:24.948085 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:01:24.948771 (+ 686us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:01:24.948994 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:01:24.949003 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:01:24.949876 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:01:24.949877 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:01:24.951043 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:01:24.951046 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:01:24.951432 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:01:24.951433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:01:24.951567 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:24.952042 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:01:24.952053 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:01:24.953620 (+ 1567us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:01:24.955191 (+ 1571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:01:24.955198 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:01:24.955200 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:01:24.955441 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:01:24.955624 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:01:24.955625 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:01:24.955626 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:01:24.955728 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:01:24.955973 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:01:24.955980 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:01:24.956116 (+ 136us) client_negotiation.cc:769] Sending connection context
0924 05:01:24.956203 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 05:01:24.956372 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":77,"threads_started":1}
I20260924 05:01:24.957011 14464 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43939
I20260924 05:01:24.957113 14464 heartbeater.cc:464] Registering TS with master...
I20260924 05:01:24.957288 14464 heartbeater.cc:511] Master 127.0.98.62:43939 requested a full tablet report, sending...
I20260924 05:01:24.957697 14139 ts_manager.cc:194] Registered new tserver with Master: fdfbafe3ba214448b087cf163e74c75a (127.0.98.2:37425)
I20260924 05:01:24.958113 14139 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50385
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 05:01:24.972939 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43939
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:01:25.056818 14469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:01:25.057029 14469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:01:25.057060 14469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:01:25.058277 14469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:01:25.058323 14469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:01:25.058358 14469 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 05:01:25.059818 14469 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43939
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14469
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:01:25.060061 14469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:01:25.060276 14469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:01:25.065276 14475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:01:25.065333 14474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:01:25.065368 14469 server_base.cc:1034] running on GCE node
W20260924 05:01:25.065333 14477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:01:25.065816 14469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:01:25.066046 14469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:01:25.067196 14469 hybrid_clock.cc:648] HybridClock initialized: now 1790226085067184 us; error 29 us; skew 500 ppm
Sep 24 05:01:25 dist-test-slave-70st krb5kdc[14094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226085, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:01:25.069689 14469 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 05:01:25.070153 14469 webserver.cc:460] Webserver started at http://127.0.98.3:35705/ using document root <none> and password file <none>
I20260924 05:01:25.070379 14469 fs_manager.cc:359] Metadata directory not provided
I20260924 05:01:25.070429 14469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:01:25.070554 14469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:01:25.071619 14469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-2/data/instance:
uuid: "42765acdc69b496cba88f5821ca315cc"
format_stamp: "Formatted at 2026-09-24 05:01:25 on dist-test-slave-70st"
server_key: "7bb3e281fe8192469659a4e6eb9f1c1c"
server_key_iv: "6687bc5781024942762f1338001d66cc"
server_key_version: "encryptionkey@0"
I20260924 05:01:25.071950 14469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-2/wal/instance:
uuid: "42765acdc69b496cba88f5821ca315cc"
format_stamp: "Formatted at 2026-09-24 05:01:25 on dist-test-slave-70st"
server_key: "7bb3e281fe8192469659a4e6eb9f1c1c"
server_key_iv: "6687bc5781024942762f1338001d66cc"
server_key_version: "encryptionkey@0"
I20260924 05:01:25.073249 14469 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 05:01:25.074057 14484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:01:25.074205 14469 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:01:25.074295 14469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-2/wal
uuid: "42765acdc69b496cba88f5821ca315cc"
format_stamp: "Formatted at 2026-09-24 05:01:25 on dist-test-slave-70st"
server_key: "7bb3e281fe8192469659a4e6eb9f1c1c"
server_key_iv: "6687bc5781024942762f1338001d66cc"
server_key_version: "encryptionkey@0"
I20260924 05:01:25.074362 14469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:01:25.084610 14469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:01:25.086791 14469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:01:25.086936 14469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:01:25.087189 14469 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:01:25.087508 14469 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:01:25.087543 14469 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:01:25.087570 14469 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:01:25.087594 14469 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:01:25.093062 14469 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43015
I20260924 05:01:25.093113 14597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43015 every 8 connection(s)
I20260924 05:01:25.093465 14469 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-2/data/info.pb
I20260924 05:01:25.097575 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 14469
I20260924 05:01:25.097684 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-2/wal/instance
I20260924 05:01:25.097864 392 external_mini_cluster.cc:1428] Setting key 5199c8abd4abb86cbc738eccc1b53636
Sep 24 05:01:25 dist-test-slave-70st krb5kdc[14094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226085, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:01:25.102345 14178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:25.094131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43081 (local address 127.0.98.62:43939)
0924 05:01:25.094176 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 05:01:25.094177 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:01:25.094383 (+ 206us) server_negotiation.cc:407] Connection header received
0924 05:01:25.094887 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:01:25.094888 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:01:25.094900 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:01:25.094963 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:01:25.095883 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:25.096378 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:25.097356 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:25.097660 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:25.097730 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:25.099898 (+ 2168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:01:25.099900 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:01:25.099900 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:01:25.099906 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:01:25.101144 (+ 1238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:01:25.101509 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:01:25.101510 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:01:25.101510 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:01:25.101571 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:01:25.101809 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:01:25.101810 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:01:25.101810 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:01:25.101928 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:01:25.101984 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 05:01:25.102188 (+ 204us) server_negotiation.cc:299] Negotiation successful
0924 05:01:25.102258 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 05:01:25.102490 14600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:25.094110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43939 (local address 127.0.98.3:43081)
0924 05:01:25.094262 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:25.094302 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 05:01:25.094785 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:01:25.095038 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:01:25.095042 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:01:25.095596 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:01:25.095784 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:01:25.095790 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:01:25.096459 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:01:25.096460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:01:25.097263 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:01:25.097266 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:01:25.097738 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:01:25.097739 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:01:25.097860 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:25.098448 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:01:25.098459 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:01:25.099823 (+ 1364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:01:25.101203 (+ 1380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:01:25.101208 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:01:25.101210 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:01:25.101404 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:01:25.101629 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:01:25.101629 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:01:25.101630 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:01:25.101732 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:01:25.101979 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:01:25.101982 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:01:25.102092 (+ 110us) client_negotiation.cc:769] Sending connection context
0924 05:01:25.102168 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 05:01:25.102293 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":76,"threads_started":1}
I20260924 05:01:25.102886 14598 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43939
I20260924 05:01:25.102974 14598 heartbeater.cc:464] Registering TS with master...
I20260924 05:01:25.103132 14598 heartbeater.cc:511] Master 127.0.98.62:43939 requested a full tablet report, sending...
I20260924 05:01:25.103441 14139 ts_manager.cc:194] Registered new tserver with Master: 42765acdc69b496cba88f5821ca315cc (127.0.98.3:43015)
I20260924 05:01:25.103777 14139 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:43081
I20260924 05:01:25.109468 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 05:01:25.116266 14178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:25.110713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38678 (local address 127.0.98.62:43939)
0924 05:01:25.110755 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 05:01:25.110757 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:01:25.110860 (+ 103us) server_negotiation.cc:407] Connection header received
0924 05:01:25.110914 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:01:25.110915 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:01:25.110924 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:01:25.110967 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:01:25.111469 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:25.111938 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:25.112646 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:25.112843 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:25.112900 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:25.114110 (+ 1210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:01:25.114111 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:01:25.114111 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:01:25.114117 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:01:25.115364 (+ 1247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:01:25.115706 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:01:25.115707 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:01:25.115707 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:01:25.115753 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:01:25.115885 (+ 132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:01:25.115886 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:01:25.115886 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:01:25.116009 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:01:25.116074 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 05:01:25.116152 (+ 78us) server_negotiation.cc:299] Negotiation successful
0924 05:01:25.116204 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 05:01:25.117651 14139 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38678:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 05:01:25.119149 14139 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 05:01:25.125020 14610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:25.121112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43015 (local address 127.0.0.1:52562)
0924 05:01:25.121443 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:25.121485 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 05:01:25.121548 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:01:25.121778 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:01:25.121781 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:01:25.121784 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:01:25.121897 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:01:25.121900 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:01:25.122877 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:01:25.122878 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:01:25.124042 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:01:25.124044 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:01:25.124729 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:01:25.124730 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:01:25.124812 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:25.124826 (+ 14us) client_negotiation.cc:769] Sending connection context
0924 05:01:25.124890 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 05:01:25.124942 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":66,"threads_started":1}
I20260924 05:01:25.125029 14609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:25.121260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52562 (local address 127.0.98.3:43015)
0924 05:01:25.121643 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0924 05:01:25.121646 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:01:25.121655 (+ 9us) server_negotiation.cc:407] Connection header received
0924 05:01:25.121687 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:01:25.121690 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:01:25.121704 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:01:25.121777 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:01:25.121979 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:25.122781 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:25.124122 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:25.124660 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:25.124721 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:25.124790 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 05:01:25.124911 (+ 121us) server_negotiation.cc:299] Negotiation successful
0924 05:01:25.124952 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":119,"threads_started":1}
I20260924 05:01:25.125334 14606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:25.121110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37425 (local address 127.0.0.1:58522)
0924 05:01:25.121442 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:25.121485 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 05:01:25.121548 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:01:25.121808 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:01:25.121809 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:01:25.121811 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:01:25.121880 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:01:25.121882 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:01:25.123242 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:01:25.123243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:01:25.124235 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:01:25.124237 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:01:25.125125 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:01:25.125126 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:01:25.125188 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:25.125192 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 05:01:25.125220 (+ 28us) client_negotiation.cc:240] Negotiation successful
0924 05:01:25.125260 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":57,"threads_started":1}
I20260924 05:01:25.125432 14607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:25.121311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38433 (local address 127.0.0.1:48496)
0924 05:01:25.121643 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:25.121651 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 05:01:25.121684 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:01:25.122112 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:01:25.122113 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:01:25.122115 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:01:25.122146 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:01:25.122147 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:01:25.123341 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:01:25.123341 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:01:25.124201 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:01:25.124203 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:01:25.125177 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:01:25.125178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:01:25.125228 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:25.125231 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:01:25.125280 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 05:01:25.125322 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":73,"threads_started":1}
I20260924 05:01:25.125466 14611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:25.121461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48496 (local address 127.0.98.1:38433)
0924 05:01:25.121689 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0924 05:01:25.121694 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 05:01:25.121909 (+ 215us) server_negotiation.cc:407] Connection header received
0924 05:01:25.121955 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:01:25.121958 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:01:25.121973 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:01:25.122040 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:01:25.122226 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:25.123258 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:25.124282 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:25.125029 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:25.125096 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:25.125151 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 05:01:25.125312 (+ 161us) server_negotiation.cc:299] Negotiation successful
0924 05:01:25.125362 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":70,"threads_started":1}
I20260924 05:01:25.126070 14608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:25.121221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58522 (local address 127.0.98.2:37425)
0924 05:01:25.121535 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0924 05:01:25.121541 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 05:01:25.121552 (+ 11us) server_negotiation.cc:407] Connection header received
0924 05:01:25.121648 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:01:25.121651 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:01:25.121667 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:01:25.121955 (+ 288us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:01:25.122027 (+ 72us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:25.123147 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:25.124316 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:25.125029 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:25.125862 (+ 833us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:25.125926 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 05:01:25.125950 (+ 24us) server_negotiation.cc:299] Negotiation successful
0924 05:01:25.125990 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":87,"threads_started":1}
I20260924 05:01:25.126008 14264 tablet_service.cc:1467] Processing CreateTablet for tablet f797a8490a3f426ea090c52e888179b6 (DEFAULT_TABLE table=test-table [id=62f49ff739f54881a9c42822e8509a86]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:01:25.126039 14532 tablet_service.cc:1467] Processing CreateTablet for tablet f797a8490a3f426ea090c52e888179b6 (DEFAULT_TABLE table=test-table [id=62f49ff739f54881a9c42822e8509a86]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:01:25.126379 14532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f797a8490a3f426ea090c52e888179b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:01:25.126385 14264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f797a8490a3f426ea090c52e888179b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:01:25.126784 14398 tablet_service.cc:1467] Processing CreateTablet for tablet f797a8490a3f426ea090c52e888179b6 (DEFAULT_TABLE table=test-table [id=62f49ff739f54881a9c42822e8509a86]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:01:25.127070 14398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f797a8490a3f426ea090c52e888179b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:01:25.129076 14612 tablet_bootstrap.cc:492] T f797a8490a3f426ea090c52e888179b6 P c60634e3d40f4fdb9b890b2d27481143: Bootstrap starting.
I20260924 05:01:25.129536 14613 tablet_bootstrap.cc:492] T f797a8490a3f426ea090c52e888179b6 P 42765acdc69b496cba88f5821ca315cc: Bootstrap starting.
I20260924 05:01:25.129776 14612 tablet_bootstrap.cc:654] T f797a8490a3f426ea090c52e888179b6 P c60634e3d40f4fdb9b890b2d27481143: Neither blocks nor log segments found. Creating new log.
I20260924 05:01:25.130084 14614 tablet_bootstrap.cc:492] T f797a8490a3f426ea090c52e888179b6 P fdfbafe3ba214448b087cf163e74c75a: Bootstrap starting.
I20260924 05:01:25.130220 14613 tablet_bootstrap.cc:654] T f797a8490a3f426ea090c52e888179b6 P 42765acdc69b496cba88f5821ca315cc: Neither blocks nor log segments found. Creating new log.
I20260924 05:01:25.130326 14612 log.cc:824] T f797a8490a3f426ea090c52e888179b6 P c60634e3d40f4fdb9b890b2d27481143: Log is configured to *not* fsync() on all Append() calls
I20260924 05:01:25.130645 14614 tablet_bootstrap.cc:654] T f797a8490a3f426ea090c52e888179b6 P fdfbafe3ba214448b087cf163e74c75a: Neither blocks nor log segments found. Creating new log.
I20260924 05:01:25.130754 14613 log.cc:824] T f797a8490a3f426ea090c52e888179b6 P 42765acdc69b496cba88f5821ca315cc: Log is configured to *not* fsync() on all Append() calls
I20260924 05:01:25.131057 14612 tablet_bootstrap.cc:492] T f797a8490a3f426ea090c52e888179b6 P c60634e3d40f4fdb9b890b2d27481143: No bootstrap required, opened a new log
I20260924 05:01:25.131114 14612 ts_tablet_manager.cc:1397] T f797a8490a3f426ea090c52e888179b6 P c60634e3d40f4fdb9b890b2d27481143: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 05:01:25.131199 14614 log.cc:824] T f797a8490a3f426ea090c52e888179b6 P fdfbafe3ba214448b087cf163e74c75a: Log is configured to *not* fsync() on all Append() calls
I20260924 05:01:25.131482 14613 tablet_bootstrap.cc:492] T f797a8490a3f426ea090c52e888179b6 P 42765acdc69b496cba88f5821ca315cc: No bootstrap required, opened a new log
I20260924 05:01:25.131583 14613 ts_tablet_manager.cc:1397] T f797a8490a3f426ea090c52e888179b6 P 42765acdc69b496cba88f5821ca315cc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 05:01:25.131832 14614 tablet_bootstrap.cc:492] T f797a8490a3f426ea090c52e888179b6 P fdfbafe3ba214448b087cf163e74c75a: No bootstrap required, opened a new log
I20260924 05:01:25.131886 14614 ts_tablet_manager.cc:1397] T f797a8490a3f426ea090c52e888179b6 P fdfbafe3ba214448b087cf163e74c75a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 05:01:25.133075 14612 raft_consensus.cc:348] T f797a8490a3f426ea090c52e888179b6 P c60634e3d40f4fdb9b890b2d27481143 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42765acdc69b496cba88f5821ca315cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43015 } } peers { permanent_uuid: "fdfbafe3ba214448b087cf163e74c75a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37425 } } peers { permanent_uuid: "c60634e3d40f4fdb9b890b2d27481143" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38433 } }
I20260924 05:01:25.133261 14612 raft_consensus.cc:374] T f797a8490a3f426ea090c52e888179b6 P c60634e3d40f4fdb9b890b2d27481143 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:01:25.133400 14612 raft_consensus.cc:729] T f797a8490a3f426ea090c52e888179b6 P c60634e3d40f4fdb9b890b2d27481143 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c60634e3d40f4fdb9b890b2d27481143, State: Initialized, Role: FOLLOWER
I20260924 05:01:25.133517 14613 raft_consensus.cc:348] T f797a8490a3f426ea090c52e888179b6 P 42765acdc69b496cba88f5821ca315cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42765acdc69b496cba88f5821ca315cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43015 } } peers { permanent_uuid: "fdfbafe3ba214448b087cf163e74c75a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37425 } } peers { permanent_uuid: "c60634e3d40f4fdb9b890b2d27481143" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38433 } }
I20260924 05:01:25.133628 14612 consensus_queue.cc:260] T f797a8490a3f426ea090c52e888179b6 P c60634e3d40f4fdb9b890b2d27481143 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42765acdc69b496cba88f5821ca315cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43015 } } peers { permanent_uuid: "fdfbafe3ba214448b087cf163e74c75a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37425 } } peers { permanent_uuid: "c60634e3d40f4fdb9b890b2d27481143" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38433 } }
I20260924 05:01:25.133713 14613 raft_consensus.cc:374] T f797a8490a3f426ea090c52e888179b6 P 42765acdc69b496cba88f5821ca315cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:01:25.133582 14614 raft_consensus.cc:348] T f797a8490a3f426ea090c52e888179b6 P fdfbafe3ba214448b087cf163e74c75a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42765acdc69b496cba88f5821ca315cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43015 } } peers { permanent_uuid: "fdfbafe3ba214448b087cf163e74c75a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37425 } } peers { permanent_uuid: "c60634e3d40f4fdb9b890b2d27481143" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38433 } }
I20260924 05:01:25.133747 14613 raft_consensus.cc:729] T f797a8490a3f426ea090c52e888179b6 P 42765acdc69b496cba88f5821ca315cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42765acdc69b496cba88f5821ca315cc, State: Initialized, Role: FOLLOWER
I20260924 05:01:25.133756 14614 raft_consensus.cc:374] T f797a8490a3f426ea090c52e888179b6 P fdfbafe3ba214448b087cf163e74c75a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:01:25.133795 14614 raft_consensus.cc:729] T f797a8490a3f426ea090c52e888179b6 P fdfbafe3ba214448b087cf163e74c75a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdfbafe3ba214448b087cf163e74c75a, State: Initialized, Role: FOLLOWER
I20260924 05:01:25.133929 14612 ts_tablet_manager.cc:1428] T f797a8490a3f426ea090c52e888179b6 P c60634e3d40f4fdb9b890b2d27481143: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260924 05:01:25.133921 14613 consensus_queue.cc:260] T f797a8490a3f426ea090c52e888179b6 P 42765acdc69b496cba88f5821ca315cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42765acdc69b496cba88f5821ca315cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43015 } } peers { permanent_uuid: "fdfbafe3ba214448b087cf163e74c75a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37425 } } peers { permanent_uuid: "c60634e3d40f4fdb9b890b2d27481143" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38433 } }
I20260924 05:01:25.133982 14614 consensus_queue.cc:260] T f797a8490a3f426ea090c52e888179b6 P fdfbafe3ba214448b087cf163e74c75a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42765acdc69b496cba88f5821ca315cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43015 } } peers { permanent_uuid: "fdfbafe3ba214448b087cf163e74c75a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37425 } } peers { permanent_uuid: "c60634e3d40f4fdb9b890b2d27481143" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38433 } }
I20260924 05:01:25.134075 14330 heartbeater.cc:503] Master 127.0.98.62:43939 was elected leader, sending a full tablet report...
I20260924 05:01:25.134207 14613 ts_tablet_manager.cc:1428] T f797a8490a3f426ea090c52e888179b6 P 42765acdc69b496cba88f5821ca315cc: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260924 05:01:25.134281 14464 heartbeater.cc:503] Master 127.0.98.62:43939 was elected leader, sending a full tablet report...
I20260924 05:01:25.134361 14598 heartbeater.cc:503] Master 127.0.98.62:43939 was elected leader, sending a full tablet report...
I20260924 05:01:25.134547 14614 ts_tablet_manager.cc:1428] T f797a8490a3f426ea090c52e888179b6 P fdfbafe3ba214448b087cf163e74c75a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
W20260924 05:01:25.197005 14465 tablet.cc:2369] T f797a8490a3f426ea090c52e888179b6 P fdfbafe3ba214448b087cf163e74c75a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 05:01:25.297367 14331 tablet.cc:2369] T f797a8490a3f426ea090c52e888179b6 P c60634e3d40f4fdb9b890b2d27481143: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 05:01:25.344030 14599 tablet.cc:2369] T f797a8490a3f426ea090c52e888179b6 P 42765acdc69b496cba88f5821ca315cc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 05:01:25.395906 14619 raft_consensus.cc:482] T f797a8490a3f426ea090c52e888179b6 P 42765acdc69b496cba88f5821ca315cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 05:01:25.396039 14619 raft_consensus.cc:504] T f797a8490a3f426ea090c52e888179b6 P 42765acdc69b496cba88f5821ca315cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42765acdc69b496cba88f5821ca315cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43015 } } peers { permanent_uuid: "fdfbafe3ba214448b087cf163e74c75a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37425 } } peers { permanent_uuid: "c60634e3d40f4fdb9b890b2d27481143" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38433 } }
I20260924 05:01:25.396363 14619 leader_election.cc:288] T f797a8490a3f426ea090c52e888179b6 P 42765acdc69b496cba88f5821ca315cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fdfbafe3ba214448b087cf163e74c75a (127.0.98.2:37425), c60634e3d40f4fdb9b890b2d27481143 (127.0.98.1:38433)
I20260924 05:01:25.400271 14611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:25.396688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42227 (local address 127.0.98.1:38433)
0924 05:01:25.396730 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 05:01:25.396732 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:01:25.396741 (+ 9us) server_negotiation.cc:407] Connection header received
0924 05:01:25.396772 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:01:25.396773 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:01:25.396785 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:01:25.396849 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:01:25.397022 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:25.397811 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:25.399265 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:25.399842 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:25.399920 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:25.399931 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 05:01:25.400177 (+ 246us) server_negotiation.cc:299] Negotiation successful
0924 05:01:25.400215 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 05:01:25.400257 14608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:25.396652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38481 (local address 127.0.98.2:37425)
0924 05:01:25.396714 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 05:01:25.396716 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:01:25.396870 (+ 154us) server_negotiation.cc:407] Connection header received
0924 05:01:25.396911 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:01:25.396917 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:01:25.396927 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:01:25.396968 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:01:25.397093 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:25.397826 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:25.399265 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:25.399851 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:25.399920 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:25.399930 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 05:01:25.400142 (+ 212us) server_negotiation.cc:299] Negotiation successful
0924 05:01:25.400187 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 05:01:25.400295 14621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:25.396601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37425 (local address 127.0.98.3:38481)
0924 05:01:25.396787 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:25.396795 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 05:01:25.396862 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:01:25.396984 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:01:25.396985 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:01:25.396987 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:01:25.397043 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:01:25.397044 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:01:25.397901 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:01:25.397903 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:01:25.399158 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:01:25.399161 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:01:25.399944 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:01:25.399945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:01:25.400068 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:25.400073 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 05:01:25.400130 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 05:01:25.400177 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":55,"thread_start_us":97,"threads_started":1}
I20260924 05:01:25.400288 14600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:25.396589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38433 (local address 127.0.98.3:42227)
0924 05:01:25.396656 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:25.396668 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 05:01:25.396714 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:01:25.396872 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:01:25.396873 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:01:25.396875 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:01:25.396952 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:01:25.396954 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:01:25.397901 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:01:25.397903 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:01:25.399158 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:01:25.399161 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:01:25.399944 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:01:25.399945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:01:25.400074 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:25.400077 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:01:25.400130 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 05:01:25.400177 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":50}
I20260924 05:01:25.400671 14418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f797a8490a3f426ea090c52e888179b6" candidate_uuid: "42765acdc69b496cba88f5821ca315cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfbafe3ba214448b087cf163e74c75a" is_pre_election: true
I20260924 05:01:25.400678 14284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f797a8490a3f426ea090c52e888179b6" candidate_uuid: "42765acdc69b496cba88f5821ca315cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c60634e3d40f4fdb9b890b2d27481143" is_pre_election: true
I20260924 05:01:25.400833 14418 raft_consensus.cc:2454] T f797a8490a3f426ea090c52e888179b6 P fdfbafe3ba214448b087cf163e74c75a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42765acdc69b496cba88f5821ca315cc in term 0.
I20260924 05:01:25.400833 14284 raft_consensus.cc:2454] T f797a8490a3f426ea090c52e888179b6 P c60634e3d40f4fdb9b890b2d27481143 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42765acdc69b496cba88f5821ca315cc in term 0.
I20260924 05:01:25.401033 14486 leader_election.cc:302] T f797a8490a3f426ea090c52e888179b6 P 42765acdc69b496cba88f5821ca315cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42765acdc69b496cba88f5821ca315cc, fdfbafe3ba214448b087cf163e74c75a; no voters:
I20260924 05:01:25.401167 14619 raft_consensus.cc:2781] T f797a8490a3f426ea090c52e888179b6 P 42765acdc69b496cba88f5821ca315cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 05:01:25.401240 14619 raft_consensus.cc:482] T f797a8490a3f426ea090c52e888179b6 P 42765acdc69b496cba88f5821ca315cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 05:01:25.401257 14619 raft_consensus.cc:3037] T f797a8490a3f426ea090c52e888179b6 P 42765acdc69b496cba88f5821ca315cc [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:01:25.401994 14619 raft_consensus.cc:504] T f797a8490a3f426ea090c52e888179b6 P 42765acdc69b496cba88f5821ca315cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42765acdc69b496cba88f5821ca315cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43015 } } peers { permanent_uuid: "fdfbafe3ba214448b087cf163e74c75a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37425 } } peers { permanent_uuid: "c60634e3d40f4fdb9b890b2d27481143" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38433 } }
I20260924 05:01:25.402114 14619 leader_election.cc:288] T f797a8490a3f426ea090c52e888179b6 P 42765acdc69b496cba88f5821ca315cc [CANDIDATE]: Term 1 election: Requested vote from peers fdfbafe3ba214448b087cf163e74c75a (127.0.98.2:37425), c60634e3d40f4fdb9b890b2d27481143 (127.0.98.1:38433)
I20260924 05:01:25.402257 14418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f797a8490a3f426ea090c52e888179b6" candidate_uuid: "42765acdc69b496cba88f5821ca315cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfbafe3ba214448b087cf163e74c75a"
I20260924 05:01:25.402278 14284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f797a8490a3f426ea090c52e888179b6" candidate_uuid: "42765acdc69b496cba88f5821ca315cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c60634e3d40f4fdb9b890b2d27481143"
I20260924 05:01:25.402328 14418 raft_consensus.cc:3037] T f797a8490a3f426ea090c52e888179b6 P fdfbafe3ba214448b087cf163e74c75a [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:01:25.402347 14284 raft_consensus.cc:3037] T f797a8490a3f426ea090c52e888179b6 P c60634e3d40f4fdb9b890b2d27481143 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:01:25.403036 14418 raft_consensus.cc:2454] T f797a8490a3f426ea090c52e888179b6 P fdfbafe3ba214448b087cf163e74c75a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42765acdc69b496cba88f5821ca315cc in term 1.
I20260924 05:01:25.403057 14284 raft_consensus.cc:2454] T f797a8490a3f426ea090c52e888179b6 P c60634e3d40f4fdb9b890b2d27481143 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42765acdc69b496cba88f5821ca315cc in term 1.
I20260924 05:01:25.403188 14487 leader_election.cc:302] T f797a8490a3f426ea090c52e888179b6 P 42765acdc69b496cba88f5821ca315cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42765acdc69b496cba88f5821ca315cc, c60634e3d40f4fdb9b890b2d27481143; no voters:
I20260924 05:01:25.403307 14619 raft_consensus.cc:2781] T f797a8490a3f426ea090c52e888179b6 P 42765acdc69b496cba88f5821ca315cc [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:01:25.403461 14619 raft_consensus.cc:686] T f797a8490a3f426ea090c52e888179b6 P 42765acdc69b496cba88f5821ca315cc [term 1 LEADER]: Becoming Leader. State: Replica: 42765acdc69b496cba88f5821ca315cc, State: Running, Role: LEADER
I20260924 05:01:25.403550 14619 consensus_queue.cc:237] T f797a8490a3f426ea090c52e888179b6 P 42765acdc69b496cba88f5821ca315cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42765acdc69b496cba88f5821ca315cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43015 } } peers { permanent_uuid: "fdfbafe3ba214448b087cf163e74c75a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37425 } } peers { permanent_uuid: "c60634e3d40f4fdb9b890b2d27481143" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38433 } }
I20260924 05:01:25.404327 14138 catalog_manager.cc:5495] T f797a8490a3f426ea090c52e888179b6 P 42765acdc69b496cba88f5821ca315cc reported cstate change: term changed from 0 to 1, leader changed from <none> to 42765acdc69b496cba88f5821ca315cc (127.0.98.3). New cstate: current_term: 1 leader_uuid: "42765acdc69b496cba88f5821ca315cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42765acdc69b496cba88f5821ca315cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43015 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdfbafe3ba214448b087cf163e74c75a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c60634e3d40f4fdb9b890b2d27481143" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38433 } health_report { overall_health: UNKNOWN } } }
I20260924 05:01:25.477684 14609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:25.475450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52578 (local address 127.0.98.3:43015)
0924 05:01:25.475503 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 05:01:25.475505 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:01:25.475513 (+ 8us) server_negotiation.cc:407] Connection header received
0924 05:01:25.475549 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:01:25.475550 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:01:25.475569 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:01:25.475631 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:01:25.475737 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:25.476207 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:25.476985 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:25.477173 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:25.477227 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:25.477326 (+ 99us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:01:25.477488 (+ 162us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:01:25.477553 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 05:01:25.477567 (+ 14us) server_negotiation.cc:299] Negotiation successful
0924 05:01:25.477585 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 05:01:25.479386 14418 raft_consensus.cc:1261] T f797a8490a3f426ea090c52e888179b6 P fdfbafe3ba214448b087cf163e74c75a [term 1 FOLLOWER]: Refusing update from remote peer 42765acdc69b496cba88f5821ca315cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:01:25.479409 14284 raft_consensus.cc:1261] T f797a8490a3f426ea090c52e888179b6 P c60634e3d40f4fdb9b890b2d27481143 [term 1 FOLLOWER]: Refusing update from remote peer 42765acdc69b496cba88f5821ca315cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:01:25.479715 14619 consensus_queue.cc:1035] T f797a8490a3f426ea090c52e888179b6 P 42765acdc69b496cba88f5821ca315cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c60634e3d40f4fdb9b890b2d27481143" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:01:25.479851 14622 consensus_queue.cc:1035] T f797a8490a3f426ea090c52e888179b6 P 42765acdc69b496cba88f5821ca315cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfbafe3ba214448b087cf163e74c75a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:01:25.482496 14627 mvcc.cc:205] Tried to move back new op lower bound from 7332766046121033728 to 7332766045813469184. Current Snapshot: MvccSnapshot[applied={T|T < 7332766046121033728 or (T in {7332766046121033728})}]
I20260924 05:01:25.484134 14139 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38678:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:01:25.484233 14139 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38678:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:01:25.484935 14139 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7725fcfb576b446cafbc9944edacc6b4: Sending DeleteTablet for 3 replicas of tablet f797a8490a3f426ea090c52e888179b6
I20260924 05:01:25.485224 14398 tablet_service.cc:1514] Processing DeleteTablet for tablet f797a8490a3f426ea090c52e888179b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:01:25 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58522
I20260924 05:01:25.485226 14532 tablet_service.cc:1514] Processing DeleteTablet for tablet f797a8490a3f426ea090c52e888179b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:01:25 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52562
I20260924 05:01:25.485498 14634 tablet_replica.cc:330] T f797a8490a3f426ea090c52e888179b6 P 42765acdc69b496cba88f5821ca315cc: stopping tablet replica
I20260924 05:01:25.485513 14264 tablet_service.cc:1514] Processing DeleteTablet for tablet f797a8490a3f426ea090c52e888179b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:01:25 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48496
I20260924 05:01:25.485620 14634 raft_consensus.cc:2229] T f797a8490a3f426ea090c52e888179b6 P 42765acdc69b496cba88f5821ca315cc [term 1 LEADER]: Raft consensus shutting down.
I20260924 05:01:25.485704 14636 tablet_replica.cc:330] T f797a8490a3f426ea090c52e888179b6 P c60634e3d40f4fdb9b890b2d27481143: stopping tablet replica
I20260924 05:01:25.485764 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 14201
I20260924 05:01:25.485803 14634 raft_consensus.cc:2258] T f797a8490a3f426ea090c52e888179b6 P 42765acdc69b496cba88f5821ca315cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:01:25.485831 14636 raft_consensus.cc:2229] T f797a8490a3f426ea090c52e888179b6 P c60634e3d40f4fdb9b890b2d27481143 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:01:25.485935 14636 raft_consensus.cc:2258] T f797a8490a3f426ea090c52e888179b6 P c60634e3d40f4fdb9b890b2d27481143 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:01:25.485931 14635 tablet_replica.cc:330] T f797a8490a3f426ea090c52e888179b6 P fdfbafe3ba214448b087cf163e74c75a: stopping tablet replica
I20260924 05:01:25.486009 14635 raft_consensus.cc:2229] T f797a8490a3f426ea090c52e888179b6 P fdfbafe3ba214448b087cf163e74c75a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:01:25.486091 14635 raft_consensus.cc:2258] T f797a8490a3f426ea090c52e888179b6 P fdfbafe3ba214448b087cf163e74c75a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:01:25.486755 14634 ts_tablet_manager.cc:1905] T f797a8490a3f426ea090c52e888179b6 P 42765acdc69b496cba88f5821ca315cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:01:25.488566 14635 ts_tablet_manager.cc:1905] T f797a8490a3f426ea090c52e888179b6 P fdfbafe3ba214448b087cf163e74c75a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:01:25.488646 14634 ts_tablet_manager.cc:1918] T f797a8490a3f426ea090c52e888179b6 P 42765acdc69b496cba88f5821ca315cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 05:01:25.488710 14634 log.cc:1196] T f797a8490a3f426ea090c52e888179b6 P 42765acdc69b496cba88f5821ca315cc: Deleting WAL directory at /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-2/wal/wals/f797a8490a3f426ea090c52e888179b6
I20260924 05:01:25.488963 14634 ts_tablet_manager.cc:1939] T f797a8490a3f426ea090c52e888179b6 P 42765acdc69b496cba88f5821ca315cc: Deleting consensus metadata
I20260924 05:01:25.489399 14125 catalog_manager.cc:4841] TS 42765acdc69b496cba88f5821ca315cc (127.0.98.3:43015): tablet f797a8490a3f426ea090c52e888179b6 (table test-table [id=62f49ff739f54881a9c42822e8509a86]) successfully deleted
I20260924 05:01:25.490180 14635 ts_tablet_manager.cc:1918] T f797a8490a3f426ea090c52e888179b6 P fdfbafe3ba214448b087cf163e74c75a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 05:01:25.490249 14635 log.cc:1196] T f797a8490a3f426ea090c52e888179b6 P fdfbafe3ba214448b087cf163e74c75a: Deleting WAL directory at /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-1/wal/wals/f797a8490a3f426ea090c52e888179b6
I20260924 05:01:25.490453 14635 ts_tablet_manager.cc:1939] T f797a8490a3f426ea090c52e888179b6 P fdfbafe3ba214448b087cf163e74c75a: Deleting consensus metadata
I20260924 05:01:25.490967 14124 catalog_manager.cc:4841] TS fdfbafe3ba214448b087cf163e74c75a (127.0.98.2:37425): tablet f797a8490a3f426ea090c52e888179b6 (table test-table [id=62f49ff739f54881a9c42822e8509a86]) successfully deleted
W20260924 05:01:25.492278 14125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38433 (error 108)
I20260924 05:01:25.492740 14610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:25.492615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38433 (local address 127.0.0.1:48500)
0924 05:01:25.492663 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:25.492697 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260924 05:01:25.492802 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 14335
W20260924 05:01:25.492856 14125 catalog_manager.cc:4584] TS c60634e3d40f4fdb9b890b2d27481143 (127.0.98.1:38433): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f797a8490a3f426ea090c52e888179b6: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38433: connect: Connection refused (error 111)
I20260924 05:01:25.497511 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 14469
I20260924 05:01:25.501762 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 14107
I20260924 05:01:25.505931 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43939
--webserver_interface=127.0.98.62
--webserver_port=34763
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43939
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:01:25.584453 14638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:01:25.584666 14638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:01:25.584688 14638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:01:25.585945 14638 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:01:25.585999 14638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:01:25.586014 14638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:01:25.586024 14638 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:01:25.586037 14638 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:01:25.587584 14638 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43939
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43939
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=34763
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14638
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:01:25.587816 14638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:01:25.588029 14638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:01:25.595747 14645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:01:25.595765 14647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:01:25.595851 14638 server_base.cc:1034] running on GCE node
W20260924 05:01:25.595762 14644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:01:25.596235 14638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:01:25.596532 14638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:01:25.597666 14638 hybrid_clock.cc:648] HybridClock initialized: now 1790226085597646 us; error 31 us; skew 500 ppm
Sep 24 05:01:25 dist-test-slave-70st krb5kdc[14094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226085, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:01:25.606531 14638 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 05:01:25.606966 14638 webserver.cc:460] Webserver started at http://127.0.98.62:34763/ using document root <none> and password file <none>
I20260924 05:01:25.607192 14638 fs_manager.cc:359] Metadata directory not provided
I20260924 05:01:25.607252 14638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:01:25.608577 14638 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 05:01:25.609139 14654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:01:25.609290 14638 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:01:25.609375 14638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/master-0/wal
uuid: "7725fcfb576b446cafbc9944edacc6b4"
format_stamp: "Formatted at 2026-09-24 05:01:24 on dist-test-slave-70st"
server_key: "d37b7160a44d8360610ee8ce5d87c5d9"
server_key_iv: "34c8bfdc26c228542a257500188c6491"
server_key_version: "encryptionkey@0"
I20260924 05:01:25.609836 14638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:01:25.627920 14638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:01:25.629751 14638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:01:25.629864 14638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:01:25.633538 14638 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43939
I20260924 05:01:25.633589 14706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43939 every 8 connection(s)
I20260924 05:01:25.633960 14638 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:01:25.635502 14707 sys_catalog.cc:263] Verifying existing consensus state
I20260924 05:01:25.636153 14707 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7725fcfb576b446cafbc9944edacc6b4: Bootstrap starting.
I20260924 05:01:25.640417 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 14638
I20260924 05:01:25.640672 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:38433
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:43939
--webserver_port=39289
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 05:01:25.644064 14707 log.cc:824] T 00000000000000000000000000000000 P 7725fcfb576b446cafbc9944edacc6b4: Log is configured to *not* fsync() on all Append() calls
I20260924 05:01:25.645949 14707 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7725fcfb576b446cafbc9944edacc6b4: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 05:01:25.646189 14707 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7725fcfb576b446cafbc9944edacc6b4: Bootstrap complete.
I20260924 05:01:25.648275 14707 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7725fcfb576b446cafbc9944edacc6b4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7725fcfb576b446cafbc9944edacc6b4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43939 } }
I20260924 05:01:25.648499 14707 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7725fcfb576b446cafbc9944edacc6b4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7725fcfb576b446cafbc9944edacc6b4, State: Initialized, Role: FOLLOWER
I20260924 05:01:25.648653 14707 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7725fcfb576b446cafbc9944edacc6b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7725fcfb576b446cafbc9944edacc6b4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43939 } }
I20260924 05:01:25.648722 14707 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7725fcfb576b446cafbc9944edacc6b4 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:01:25.648775 14707 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7725fcfb576b446cafbc9944edacc6b4 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:01:25.648821 14707 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7725fcfb576b446cafbc9944edacc6b4 [term 1 FOLLOWER]: Advancing to term 2
I20260924 05:01:25.649820 14707 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7725fcfb576b446cafbc9944edacc6b4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7725fcfb576b446cafbc9944edacc6b4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43939 } }
I20260924 05:01:25.649920 14707 leader_election.cc:302] T 00000000000000000000000000000000 P 7725fcfb576b446cafbc9944edacc6b4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7725fcfb576b446cafbc9944edacc6b4; no voters:
I20260924 05:01:25.650069 14707 leader_election.cc:288] T 00000000000000000000000000000000 P 7725fcfb576b446cafbc9944edacc6b4 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 05:01:25.650115 14711 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7725fcfb576b446cafbc9944edacc6b4 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 05:01:25.650259 14711 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7725fcfb576b446cafbc9944edacc6b4 [term 2 LEADER]: Becoming Leader. State: Replica: 7725fcfb576b446cafbc9944edacc6b4, State: Running, Role: LEADER
I20260924 05:01:25.650326 14707 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7725fcfb576b446cafbc9944edacc6b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:01:25.650425 14711 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7725fcfb576b446cafbc9944edacc6b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7725fcfb576b446cafbc9944edacc6b4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43939 } }
I20260924 05:01:25.650707 14712 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7725fcfb576b446cafbc9944edacc6b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "7725fcfb576b446cafbc9944edacc6b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7725fcfb576b446cafbc9944edacc6b4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43939 } } }
I20260924 05:01:25.650775 14712 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7725fcfb576b446cafbc9944edacc6b4 [sys.catalog]: This master's current role is: LEADER
I20260924 05:01:25.650970 14713 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7725fcfb576b446cafbc9944edacc6b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7725fcfb576b446cafbc9944edacc6b4. Latest consensus state: current_term: 2 leader_uuid: "7725fcfb576b446cafbc9944edacc6b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7725fcfb576b446cafbc9944edacc6b4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43939 } } }
I20260924 05:01:25.651077 14713 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7725fcfb576b446cafbc9944edacc6b4 [sys.catalog]: This master's current role is: LEADER
I20260924 05:01:25.651189 14716 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:01:25.651937 14716 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:01:25.652079 14716 catalog_manager.cc:1259] Loaded cluster ID: 46f0a1bdd2da4de58d54f3cb3ba37607
I20260924 05:01:25.652280 14716 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:01:25.652529 14716 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260924 05:01:25.657102 14716 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260924 05:01:25.659547 14716 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:01:25.659756 14716 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7725fcfb576b446cafbc9944edacc6b4: Loaded TSK: 0
I20260924 05:01:25.659978 14716 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 05:01:25.721218 14709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:01:25.721446 14709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:01:25.721480 14709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:01:25.722605 14709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:01:25.722644 14709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:01:25.722678 14709 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 05:01:25.724088 14709 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:38433
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=39289
--tserver_master_addrs=127.0.98.62:43939
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14709
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:01:25.724314 14709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:01:25.724524 14709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:01:25.729326 14734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:01:25.729403 14737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:01:25.729380 14735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:01:25.729485 14709 server_base.cc:1034] running on GCE node
I20260924 05:01:25.729719 14709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:01:25.729949 14709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:01:25.731092 14709 hybrid_clock.cc:648] HybridClock initialized: now 1790226085731072 us; error 35 us; skew 500 ppm
Sep 24 05:01:25 dist-test-slave-70st krb5kdc[14094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226085, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:01:25.733601 14709 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 05:01:25.734045 14709 webserver.cc:460] Webserver started at http://127.0.98.1:39289/ using document root <none> and password file <none>
I20260924 05:01:25.734261 14709 fs_manager.cc:359] Metadata directory not provided
I20260924 05:01:25.734321 14709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:01:25.735673 14709 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 05:01:25.736274 14744 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:01:25.736462 14709 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:01:25.736557 14709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-0/wal
uuid: "c60634e3d40f4fdb9b890b2d27481143"
format_stamp: "Formatted at 2026-09-24 05:01:24 on dist-test-slave-70st"
server_key: "57b7b43a095930ff567c4befa65feeb5"
server_key_iv: "049cc7180b3f42ed314190f68ff24118"
server_key_version: "encryptionkey@0"
I20260924 05:01:25.736989 14709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:01:25.758196 14709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:01:25.760566 14709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:01:25.760718 14709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:01:25.760988 14709 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:01:25.761615 14751 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 05:01:25.763083 14709 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 05:01:25.763144 14709 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260924 05:01:25.763180 14709 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 05:01:25.764034 14709 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 05:01:25.764076 14709 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260924 05:01:25.764158 14751 tablet_bootstrap.cc:492] T f797a8490a3f426ea090c52e888179b6 P c60634e3d40f4fdb9b890b2d27481143: Bootstrap starting.
I20260924 05:01:25.770401 14709 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38433
I20260924 05:01:25.770627 14858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38433 every 8 connection(s)
I20260924 05:01:25.770864 14709 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-0/data/info.pb
I20260924 05:01:25.774744 14751 log.cc:824] T f797a8490a3f426ea090c52e888179b6 P c60634e3d40f4fdb9b890b2d27481143: Log is configured to *not* fsync() on all Append() calls
I20260924 05:01:25.775431 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 14709
I20260924 05:01:25.775717 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:37425
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:43939
--webserver_port=46749
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 05:01:25.778986 14751 tablet_bootstrap.cc:492] T f797a8490a3f426ea090c52e888179b6 P c60634e3d40f4fdb9b890b2d27481143: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 05:01:25.779311 14751 tablet_bootstrap.cc:492] T f797a8490a3f426ea090c52e888179b6 P c60634e3d40f4fdb9b890b2d27481143: Bootstrap complete.
Sep 24 05:01:25 dist-test-slave-70st krb5kdc[14094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226085, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:01:25.780450 14751 ts_tablet_manager.cc:1397] T f797a8490a3f426ea090c52e888179b6 P c60634e3d40f4fdb9b890b2d27481143: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.004s
I20260924 05:01:25.781953 14751 raft_consensus.cc:348] T f797a8490a3f426ea090c52e888179b6 P c60634e3d40f4fdb9b890b2d27481143 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42765acdc69b496cba88f5821ca315cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43015 } } peers { permanent_uuid: "fdfbafe3ba214448b087cf163e74c75a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37425 } } peers { permanent_uuid: "c60634e3d40f4fdb9b890b2d27481143" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38433 } }
I20260924 05:01:25.782394 14751 raft_consensus.cc:729] T f797a8490a3f426ea090c52e888179b6 P c60634e3d40f4fdb9b890b2d27481143 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c60634e3d40f4fdb9b890b2d27481143, State: Initialized, Role: FOLLOWER
I20260924 05:01:25.782625 14751 consensus_queue.cc:260] T f797a8490a3f426ea090c52e888179b6 P c60634e3d40f4fdb9b890b2d27481143 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42765acdc69b496cba88f5821ca315cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43015 } } peers { permanent_uuid: "fdfbafe3ba214448b087cf163e74c75a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37425 } } peers { permanent_uuid: "c60634e3d40f4fdb9b890b2d27481143" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38433 } }
I20260924 05:01:25.783010 14751 ts_tablet_manager.cc:1428] T f797a8490a3f426ea090c52e888179b6 P c60634e3d40f4fdb9b890b2d27481143: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 05:01:25.784066 14862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:25.771730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56195 (local address 127.0.98.62:43939)
0924 05:01:25.772008 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0924 05:01:25.772018 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 05:01:25.772082 (+ 64us) server_negotiation.cc:407] Connection header received
0924 05:01:25.772764 (+ 682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:01:25.772773 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:01:25.772846 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:01:25.773049 (+ 203us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:01:25.773836 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:25.775265 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:25.776732 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:25.777466 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:25.778062 (+ 596us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:25.780304 (+ 2242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:01:25.780308 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:01:25.780308 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:01:25.780320 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:01:25.782225 (+ 1905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:01:25.782696 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:01:25.782699 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:01:25.782699 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:01:25.782800 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:01:25.783040 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:01:25.783040 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:01:25.783041 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:01:25.783297 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:01:25.783378 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 05:01:25.783738 (+ 360us) server_negotiation.cc:299] Negotiation successful
0924 05:01:25.783869 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":75,"threads_started":1}
I20260924 05:01:25.784077 14861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:25.771638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43939 (local address 127.0.98.1:56195)
0924 05:01:25.771852 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:25.771900 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 05:01:25.772514 (+ 614us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:01:25.773068 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:01:25.773072 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:01:25.773410 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:01:25.773688 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:01:25.773694 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:01:25.775383 (+ 1689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:01:25.775384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:01:25.776634 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:01:25.776637 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:01:25.777590 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:01:25.777592 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:01:25.777744 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:25.778418 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:01:25.778428 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:01:25.780198 (+ 1770us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:01:25.782316 (+ 2118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:01:25.782322 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:01:25.782324 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:01:25.782612 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:01:25.782867 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:01:25.782867 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:01:25.782868 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:01:25.782977 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:01:25.783378 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:01:25.783382 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:01:25.783552 (+ 170us) client_negotiation.cc:769] Sending connection context
0924 05:01:25.783683 (+ 131us) client_negotiation.cc:240] Negotiation successful
0924 05:01:25.783878 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":118,"threads_started":1}
I20260924 05:01:25.785154 14859 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43939
I20260924 05:01:25.785256 14859 heartbeater.cc:464] Registering TS with master...
I20260924 05:01:25.785475 14859 heartbeater.cc:511] Master 127.0.98.62:43939 requested a full tablet report, sending...
I20260924 05:01:25.786293 14671 ts_manager.cc:194] Registered new tserver with Master: c60634e3d40f4fdb9b890b2d27481143 (127.0.98.1:38433)
I20260924 05:01:25.787540 14671 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:56195
I20260924 05:01:25.788509 14859 heartbeater.cc:503] Master 127.0.98.62:43939 was elected leader, sending a full tablet report...
I20260924 05:01:25.791945 14867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:25.787297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38433 (local address 127.0.0.1:50174)
0924 05:01:25.788333 (+ 1036us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:25.788362 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0924 05:01:25.788433 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:01:25.788648 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:01:25.788651 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:01:25.788653 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:01:25.788712 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:01:25.788715 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:01:25.789992 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:01:25.789993 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:01:25.790821 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:01:25.790824 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:01:25.791714 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:01:25.791715 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:01:25.791777 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:25.791790 (+ 13us) client_negotiation.cc:769] Sending connection context
0924 05:01:25.791848 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 05:01:25.791888 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1015,"thread_start_us":75,"threads_started":1}
I20260924 05:01:25.791972 14866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:25.787124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50174 (local address 127.0.98.1:38433)
0924 05:01:25.788282 (+ 1158us) server_negotiation.cc:206] Beginning negotiation
0924 05:01:25.788286 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 05:01:25.788435 (+ 149us) server_negotiation.cc:407] Connection header received
0924 05:01:25.788505 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:01:25.788507 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:01:25.788522 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:01:25.788584 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:01:25.788775 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:25.789893 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:25.790903 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:25.791626 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:25.791717 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:25.791772 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 05:01:25.791861 (+ 89us) server_negotiation.cc:299] Negotiation successful
0924 05:01:25.791899 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1131,"thread_start_us":86,"threads_started":1}
I20260924 05:01:25.792438 14793 tablet_service.cc:1514] Processing DeleteTablet for tablet f797a8490a3f426ea090c52e888179b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:01:25 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50174
I20260924 05:01:25.793047 14868 tablet_replica.cc:330] stopping tablet replica
I20260924 05:01:25.793151 14868 raft_consensus.cc:2229] T f797a8490a3f426ea090c52e888179b6 P c60634e3d40f4fdb9b890b2d27481143 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:01:25.793215 14868 raft_consensus.cc:2258] T f797a8490a3f426ea090c52e888179b6 P c60634e3d40f4fdb9b890b2d27481143 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:01:25.793535 14868 ts_tablet_manager.cc:1905] T f797a8490a3f426ea090c52e888179b6 P c60634e3d40f4fdb9b890b2d27481143: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:01:25.795499 14868 ts_tablet_manager.cc:1918] T f797a8490a3f426ea090c52e888179b6 P c60634e3d40f4fdb9b890b2d27481143: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 05:01:25.795562 14868 log.cc:1196] T f797a8490a3f426ea090c52e888179b6 P c60634e3d40f4fdb9b890b2d27481143: Deleting WAL directory at /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-0/wal/wals/f797a8490a3f426ea090c52e888179b6
I20260924 05:01:25.795758 14868 ts_tablet_manager.cc:1939] T f797a8490a3f426ea090c52e888179b6 P c60634e3d40f4fdb9b890b2d27481143: Deleting consensus metadata
I20260924 05:01:25.796212 14657 catalog_manager.cc:4841] TS c60634e3d40f4fdb9b890b2d27481143 (127.0.98.1:38433): tablet f797a8490a3f426ea090c52e888179b6 (table test-table [id=62f49ff739f54881a9c42822e8509a86]) successfully deleted
W20260924 05:01:25.862036 14864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:01:25.862237 14864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:01:25.862259 14864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:01:25.863333 14864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:01:25.863368 14864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:01:25.863390 14864 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 05:01:25.864732 14864 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:37425
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=46749
--tserver_master_addrs=127.0.98.62:43939
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14864
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:01:25.864908 14864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:01:25.865079 14864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:01:25.870157 14874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:01:25.870144 14876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:01:25.870144 14873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:01:25.870316 14864 server_base.cc:1034] running on GCE node
I20260924 05:01:25.870568 14864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:01:25.870761 14864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:01:25.871920 14864 hybrid_clock.cc:648] HybridClock initialized: now 1790226085871905 us; error 28 us; skew 500 ppm
Sep 24 05:01:25 dist-test-slave-70st krb5kdc[14094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226085, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:01:25.874382 14864 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 05:01:25.874780 14864 webserver.cc:460] Webserver started at http://127.0.98.2:46749/ using document root <none> and password file <none>
I20260924 05:01:25.874982 14864 fs_manager.cc:359] Metadata directory not provided
I20260924 05:01:25.875042 14864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:01:25.876333 14864 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 05:01:25.876950 14883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:01:25.877117 14864 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 05:01:25.877203 14864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-1/wal
uuid: "fdfbafe3ba214448b087cf163e74c75a"
format_stamp: "Formatted at 2026-09-24 05:01:24 on dist-test-slave-70st"
server_key: "d8df35ec213e6c68fead8d0c0c14d84c"
server_key_iv: "781a03fd7a040819393296fa0b07da22"
server_key_version: "encryptionkey@0"
I20260924 05:01:25.877609 14864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:01:25.886590 14864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:01:25.888435 14864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:01:25.888571 14864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:01:25.888788 14864 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:01:25.889089 14864 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:01:25.889124 14864 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:01:25.889150 14864 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:01:25.889171 14864 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:01:25.894377 14864 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37425
I20260924 05:01:25.894439 14996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37425 every 8 connection(s)
I20260924 05:01:25.894732 14864 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-1/data/info.pb
Sep 24 05:01:25 dist-test-slave-70st krb5kdc[14094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226085, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:01:25.902575 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 14864
I20260924 05:01:25.902864 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:43015
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:43939
--webserver_port=35705
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 05:01:25.903321 14862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:25.895373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51489 (local address 127.0.98.62:43939)
0924 05:01:25.895425 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 05:01:25.895427 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:01:25.895627 (+ 200us) server_negotiation.cc:407] Connection header received
0924 05:01:25.896143 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:01:25.896144 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:01:25.896161 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:01:25.896214 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:01:25.897136 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:25.897691 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:25.898643 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:25.898842 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:25.898897 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:25.900875 (+ 1978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:01:25.900877 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:01:25.900877 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:01:25.900884 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:01:25.902048 (+ 1164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:01:25.902417 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:01:25.902418 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:01:25.902419 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:01:25.902473 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:01:25.902677 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:01:25.902678 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:01:25.902678 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:01:25.902796 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:01:25.902852 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 05:01:25.903138 (+ 286us) server_negotiation.cc:299] Negotiation successful
0924 05:01:25.903227 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 05:01:25.903474 14999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:25.895364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43939 (local address 127.0.98.2:51489)
0924 05:01:25.895518 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:25.895552 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 05:01:25.896042 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:01:25.896288 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:01:25.896292 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:01:25.896850 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:01:25.897040 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:01:25.897045 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:01:25.897773 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:01:25.897774 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:01:25.898561 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:01:25.898564 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:01:25.898909 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:01:25.898910 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:01:25.899025 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:25.899467 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:01:25.899476 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:01:25.900789 (+ 1313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:01:25.902121 (+ 1332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:01:25.902127 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:01:25.902130 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:01:25.902349 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:01:25.902526 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:01:25.902526 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:01:25.902527 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:01:25.902623 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:01:25.902851 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:01:25.902858 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:01:25.903007 (+ 149us) client_negotiation.cc:769] Sending connection context
0924 05:01:25.903105 (+ 98us) client_negotiation.cc:240] Negotiation successful
0924 05:01:25.903263 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":73,"threads_started":1}
I20260924 05:01:25.903904 14997 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43939
I20260924 05:01:25.903995 14997 heartbeater.cc:464] Registering TS with master...
I20260924 05:01:25.904165 14997 heartbeater.cc:511] Master 127.0.98.62:43939 requested a full tablet report, sending...
I20260924 05:01:25.904536 14671 ts_manager.cc:194] Registered new tserver with Master: fdfbafe3ba214448b087cf163e74c75a (127.0.98.2:37425)
I20260924 05:01:25.904881 14671 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:51489
W20260924 05:01:25.983222 15000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:01:25.983435 15000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:01:25.983469 15000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:01:25.984586 15000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:01:25.984633 15000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:01:25.984670 15000 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 05:01:25.986150 15000 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:43015
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=35705
--tserver_master_addrs=127.0.98.62:43939
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15000
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:01:25.986399 15000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:01:25.986613 15000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:01:25.991448 15008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:01:25.991465 15005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:01:25.991539 15006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:01:25.991518 15000 server_base.cc:1034] running on GCE node
I20260924 05:01:25.991832 15000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:01:25.992098 15000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:01:25.993269 15000 hybrid_clock.cc:648] HybridClock initialized: now 1790226085993247 us; error 36 us; skew 500 ppm
Sep 24 05:01:25 dist-test-slave-70st krb5kdc[14094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226085, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:01:25.995582 15000 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 05:01:25.995985 15000 webserver.cc:460] Webserver started at http://127.0.98.3:35705/ using document root <none> and password file <none>
I20260924 05:01:25.996194 15000 fs_manager.cc:359] Metadata directory not provided
I20260924 05:01:25.996264 15000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:01:25.997634 15000 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 05:01:25.998178 15015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:01:25.998315 15000 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 05:01:25.998399 15000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-2/wal
uuid: "42765acdc69b496cba88f5821ca315cc"
format_stamp: "Formatted at 2026-09-24 05:01:25 on dist-test-slave-70st"
server_key: "7bb3e281fe8192469659a4e6eb9f1c1c"
server_key_iv: "6687bc5781024942762f1338001d66cc"
server_key_version: "encryptionkey@0"
I20260924 05:01:25.998724 15000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:01:26.016413 15000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:01:26.018569 15000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:01:26.018718 15000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:01:26.018949 15000 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:01:26.019265 15000 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:01:26.019302 15000 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:01:26.019348 15000 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:01:26.019377 15000 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:01:26.024920 15000 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43015
I20260924 05:01:26.024979 15128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43015 every 8 connection(s)
I20260924 05:01:26.025292 15000 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-2/data/info.pb
I20260924 05:01:26.027752 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 15000
Sep 24 05:01:26 dist-test-slave-70st krb5kdc[14094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226085, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:01:26.034082 15133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:26.028102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43542 (local address 127.0.98.62:43939)
0924 05:01:26.028210 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0924 05:01:26.028212 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:01:26.028275 (+ 63us) server_negotiation.cc:407] Connection header received
0924 05:01:26.028480 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:01:26.028481 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:01:26.028494 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:01:26.028550 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:01:26.029096 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:26.029629 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:26.030307 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:26.030502 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:26.030554 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:26.031814 (+ 1260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:01:26.031815 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:01:26.031816 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:01:26.031822 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:01:26.033070 (+ 1248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:01:26.033357 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:01:26.033358 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:01:26.033358 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:01:26.033445 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:01:26.033603 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:01:26.033604 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:01:26.033604 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:01:26.033709 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:01:26.033870 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0924 05:01:26.033906 (+ 36us) server_negotiation.cc:299] Negotiation successful
0924 05:01:26.033978 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":88,"thread_start_us":53,"threads_started":1}
I20260924 05:01:26.034545 14862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:26.025976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45343 (local address 127.0.98.62:43939)
0924 05:01:26.026020 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 05:01:26.026022 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:01:26.026243 (+ 221us) server_negotiation.cc:407] Connection header received
0924 05:01:26.026742 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:01:26.026743 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:01:26.026756 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:01:26.026809 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:01:26.027718 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:26.028298 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:26.029459 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:26.029669 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:26.029723 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:26.031937 (+ 2214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:01:26.031939 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:01:26.031939 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:01:26.031945 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:01:26.033234 (+ 1289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:01:26.033621 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:01:26.033622 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:01:26.033623 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:01:26.033675 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:01:26.033896 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:01:26.033896 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:01:26.033897 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:01:26.034004 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:01:26.034056 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 05:01:26.034398 (+ 342us) server_negotiation.cc:299] Negotiation successful
0924 05:01:26.034470 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":20}
I20260924 05:01:26.034763 15131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:26.025961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43939 (local address 127.0.98.3:45343)
0924 05:01:26.026114 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:26.026149 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 05:01:26.026642 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:01:26.026879 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:01:26.026883 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:01:26.027432 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:01:26.027617 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:01:26.027622 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:01:26.028474 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:01:26.028475 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:01:26.029367 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:01:26.029369 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:01:26.029744 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:01:26.029744 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:01:26.029878 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:26.030344 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:01:26.030353 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:01:26.031843 (+ 1490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:01:26.033303 (+ 1460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:01:26.033308 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:01:26.033310 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:01:26.033560 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:01:26.033726 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:01:26.033727 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:01:26.033727 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:01:26.033836 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:01:26.034082 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:01:26.034093 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:01:26.034256 (+ 163us) client_negotiation.cc:769] Sending connection context
0924 05:01:26.034378 (+ 122us) client_negotiation.cc:240] Negotiation successful
0924 05:01:26.034535 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":76,"threads_started":1}
I20260924 05:01:26.035229 15129 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43939
I20260924 05:01:26.035312 15129 heartbeater.cc:464] Registering TS with master...
I20260924 05:01:26.035477 15129 heartbeater.cc:511] Master 127.0.98.62:43939 requested a full tablet report, sending...
I20260924 05:01:26.035789 14671 ts_manager.cc:194] Registered new tserver with Master: 42765acdc69b496cba88f5821ca315cc (127.0.98.3:43015)
I20260924 05:01:26.036126 14671 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45343
I20260924 05:01:26.045595 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 05:01:26.048044 14862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:26.045905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43544 (local address 127.0.98.62:43939)
0924 05:01:26.045948 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 05:01:26.045949 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:01:26.046039 (+ 90us) server_negotiation.cc:407] Connection header received
0924 05:01:26.046095 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:01:26.046096 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:01:26.046128 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:01:26.046187 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:01:26.046273 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:26.046728 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:26.047477 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:26.047657 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:26.047703 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:26.047807 (+ 104us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:01:26.047906 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:01:26.047964 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 05:01:26.047977 (+ 13us) server_negotiation.cc:299] Negotiation successful
0924 05:01:26.047990 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 05:01:26.048288 14671 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:43544:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 05:01:26.049901 14671 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 05:01:26.051873 14793 tablet_service.cc:1467] Processing CreateTablet for tablet c3828df18d3a4774a0053986b0f1baae (DEFAULT_TABLE table=test-table [id=e508c6471c2047cdab1cb00e16334cb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:01:26.052060 14793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3828df18d3a4774a0053986b0f1baae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:01:26.053325 14751 tablet_bootstrap.cc:492] T c3828df18d3a4774a0053986b0f1baae P c60634e3d40f4fdb9b890b2d27481143: Bootstrap starting.
I20260924 05:01:26.053709 14751 tablet_bootstrap.cc:654] T c3828df18d3a4774a0053986b0f1baae P c60634e3d40f4fdb9b890b2d27481143: Neither blocks nor log segments found. Creating new log.
I20260924 05:01:26.054250 14751 tablet_bootstrap.cc:492] T c3828df18d3a4774a0053986b0f1baae P c60634e3d40f4fdb9b890b2d27481143: No bootstrap required, opened a new log
I20260924 05:01:26.054296 14751 ts_tablet_manager.cc:1397] T c3828df18d3a4774a0053986b0f1baae P c60634e3d40f4fdb9b890b2d27481143: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 05:01:26.054399 14751 raft_consensus.cc:348] T c3828df18d3a4774a0053986b0f1baae P c60634e3d40f4fdb9b890b2d27481143 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42765acdc69b496cba88f5821ca315cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43015 } } peers { permanent_uuid: "fdfbafe3ba214448b087cf163e74c75a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37425 } } peers { permanent_uuid: "c60634e3d40f4fdb9b890b2d27481143" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38433 } }
I20260924 05:01:26.054456 14751 raft_consensus.cc:374] T c3828df18d3a4774a0053986b0f1baae P c60634e3d40f4fdb9b890b2d27481143 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:01:26.054494 14751 raft_consensus.cc:729] T c3828df18d3a4774a0053986b0f1baae P c60634e3d40f4fdb9b890b2d27481143 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c60634e3d40f4fdb9b890b2d27481143, State: Initialized, Role: FOLLOWER
I20260924 05:01:26.054530 14751 consensus_queue.cc:260] T c3828df18d3a4774a0053986b0f1baae P c60634e3d40f4fdb9b890b2d27481143 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42765acdc69b496cba88f5821ca315cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43015 } } peers { permanent_uuid: "fdfbafe3ba214448b087cf163e74c75a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37425 } } peers { permanent_uuid: "c60634e3d40f4fdb9b890b2d27481143" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38433 } }
I20260924 05:01:26.054596 14751 ts_tablet_manager.cc:1428] T c3828df18d3a4774a0053986b0f1baae P c60634e3d40f4fdb9b890b2d27481143: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 05:01:26.055876 15135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:26.051612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37425 (local address 127.0.0.1:50994)
0924 05:01:26.051853 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:26.051862 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:01:26.051905 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:01:26.052194 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:01:26.052194 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:01:26.052197 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:01:26.052239 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:01:26.052241 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:01:26.053436 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:01:26.053437 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:01:26.054325 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:01:26.054328 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:01:26.055683 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:01:26.055684 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:01:26.055736 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:26.055739 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:01:26.055775 (+ 36us) client_negotiation.cc:240] Negotiation successful
0924 05:01:26.055805 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":95,"threads_started":1}
I20260924 05:01:26.055922 15137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:26.051847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50994 (local address 127.0.98.2:37425)
0924 05:01:26.052048 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0924 05:01:26.052053 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 05:01:26.052062 (+ 9us) server_negotiation.cc:407] Connection header received
0924 05:01:26.052099 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:01:26.052102 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:01:26.052118 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:01:26.052186 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:01:26.052289 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:26.053319 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:26.054399 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:26.055611 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:26.055684 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:26.055736 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 05:01:26.055797 (+ 61us) server_negotiation.cc:299] Negotiation successful
0924 05:01:26.055837 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":96,"threads_started":1}
I20260924 05:01:26.056600 14931 tablet_service.cc:1467] Processing CreateTablet for tablet c3828df18d3a4774a0053986b0f1baae (DEFAULT_TABLE table=test-table [id=e508c6471c2047cdab1cb00e16334cb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:01:26.056799 15136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:26.051750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42794 (local address 127.0.98.3:43015)
0924 05:01:26.053394 (+ 1644us) server_negotiation.cc:206] Beginning negotiation
0924 05:01:26.053398 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 05:01:26.053409 (+ 11us) server_negotiation.cc:407] Connection header received
0924 05:01:26.053477 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:01:26.053479 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:01:26.053495 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:01:26.053582 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:01:26.053683 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:26.054637 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:26.055874 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:26.056490 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:26.056561 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:26.056613 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 05:01:26.056694 (+ 81us) server_negotiation.cc:299] Negotiation successful
0924 05:01:26.056728 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1609,"thread_start_us":105,"threads_started":1}
I20260924 05:01:26.056813 14867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:26.051574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43015 (local address 127.0.0.1:42794)
0924 05:01:26.051613 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:26.051623 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:01:26.051662 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:01:26.053586 (+ 1924us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:01:26.053587 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:01:26.053589 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:01:26.053619 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:01:26.053620 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:01:26.054762 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:01:26.054763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:01:26.055781 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:01:26.055784 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:01:26.056578 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:01:26.056579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:01:26.056628 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:26.056631 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:01:26.056680 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 05:01:26.056737 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21}
I20260924 05:01:26.056943 14931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3828df18d3a4774a0053986b0f1baae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:01:26.057471 15063 tablet_service.cc:1467] Processing CreateTablet for tablet c3828df18d3a4774a0053986b0f1baae (DEFAULT_TABLE table=test-table [id=e508c6471c2047cdab1cb00e16334cb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:01:26.057747 15063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3828df18d3a4774a0053986b0f1baae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:01:26.060256 15139 tablet_bootstrap.cc:492] T c3828df18d3a4774a0053986b0f1baae P fdfbafe3ba214448b087cf163e74c75a: Bootstrap starting.
I20260924 05:01:26.060779 15140 tablet_bootstrap.cc:492] T c3828df18d3a4774a0053986b0f1baae P 42765acdc69b496cba88f5821ca315cc: Bootstrap starting.
I20260924 05:01:26.060822 15139 tablet_bootstrap.cc:654] T c3828df18d3a4774a0053986b0f1baae P fdfbafe3ba214448b087cf163e74c75a: Neither blocks nor log segments found. Creating new log.
I20260924 05:01:26.061255 15139 log.cc:824] T c3828df18d3a4774a0053986b0f1baae P fdfbafe3ba214448b087cf163e74c75a: Log is configured to *not* fsync() on all Append() calls
I20260924 05:01:26.061338 15140 tablet_bootstrap.cc:654] T c3828df18d3a4774a0053986b0f1baae P 42765acdc69b496cba88f5821ca315cc: Neither blocks nor log segments found. Creating new log.
I20260924 05:01:26.061862 15140 log.cc:824] T c3828df18d3a4774a0053986b0f1baae P 42765acdc69b496cba88f5821ca315cc: Log is configured to *not* fsync() on all Append() calls
I20260924 05:01:26.061971 15139 tablet_bootstrap.cc:492] T c3828df18d3a4774a0053986b0f1baae P fdfbafe3ba214448b087cf163e74c75a: No bootstrap required, opened a new log
I20260924 05:01:26.062042 15139 ts_tablet_manager.cc:1397] T c3828df18d3a4774a0053986b0f1baae P fdfbafe3ba214448b087cf163e74c75a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 05:01:26.062512 15140 tablet_bootstrap.cc:492] T c3828df18d3a4774a0053986b0f1baae P 42765acdc69b496cba88f5821ca315cc: No bootstrap required, opened a new log
I20260924 05:01:26.062567 15140 ts_tablet_manager.cc:1397] T c3828df18d3a4774a0053986b0f1baae P 42765acdc69b496cba88f5821ca315cc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 05:01:26.063479 15139 raft_consensus.cc:348] T c3828df18d3a4774a0053986b0f1baae P fdfbafe3ba214448b087cf163e74c75a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42765acdc69b496cba88f5821ca315cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43015 } } peers { permanent_uuid: "fdfbafe3ba214448b087cf163e74c75a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37425 } } peers { permanent_uuid: "c60634e3d40f4fdb9b890b2d27481143" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38433 } }
I20260924 05:01:26.063650 15139 raft_consensus.cc:374] T c3828df18d3a4774a0053986b0f1baae P fdfbafe3ba214448b087cf163e74c75a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:01:26.063676 15139 raft_consensus.cc:729] T c3828df18d3a4774a0053986b0f1baae P fdfbafe3ba214448b087cf163e74c75a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdfbafe3ba214448b087cf163e74c75a, State: Initialized, Role: FOLLOWER
I20260924 05:01:26.063793 15139 consensus_queue.cc:260] T c3828df18d3a4774a0053986b0f1baae P fdfbafe3ba214448b087cf163e74c75a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42765acdc69b496cba88f5821ca315cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43015 } } peers { permanent_uuid: "fdfbafe3ba214448b087cf163e74c75a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37425 } } peers { permanent_uuid: "c60634e3d40f4fdb9b890b2d27481143" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38433 } }
I20260924 05:01:26.064044 15139 ts_tablet_manager.cc:1428] T c3828df18d3a4774a0053986b0f1baae P fdfbafe3ba214448b087cf163e74c75a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 05:01:26.064049 14997 heartbeater.cc:503] Master 127.0.98.62:43939 was elected leader, sending a full tablet report...
I20260924 05:01:26.064014 15140 raft_consensus.cc:348] T c3828df18d3a4774a0053986b0f1baae P 42765acdc69b496cba88f5821ca315cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42765acdc69b496cba88f5821ca315cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43015 } } peers { permanent_uuid: "fdfbafe3ba214448b087cf163e74c75a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37425 } } peers { permanent_uuid: "c60634e3d40f4fdb9b890b2d27481143" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38433 } }
I20260924 05:01:26.064148 15140 raft_consensus.cc:374] T c3828df18d3a4774a0053986b0f1baae P 42765acdc69b496cba88f5821ca315cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:01:26.064188 15140 raft_consensus.cc:729] T c3828df18d3a4774a0053986b0f1baae P 42765acdc69b496cba88f5821ca315cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42765acdc69b496cba88f5821ca315cc, State: Initialized, Role: FOLLOWER
I20260924 05:01:26.064288 15140 consensus_queue.cc:260] T c3828df18d3a4774a0053986b0f1baae P 42765acdc69b496cba88f5821ca315cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42765acdc69b496cba88f5821ca315cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43015 } } peers { permanent_uuid: "fdfbafe3ba214448b087cf163e74c75a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37425 } } peers { permanent_uuid: "c60634e3d40f4fdb9b890b2d27481143" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38433 } }
I20260924 05:01:26.064546 15140 ts_tablet_manager.cc:1428] T c3828df18d3a4774a0053986b0f1baae P 42765acdc69b496cba88f5821ca315cc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 05:01:26.064574 15129 heartbeater.cc:503] Master 127.0.98.62:43939 was elected leader, sending a full tablet report...
W20260924 05:01:26.145274 14998 tablet.cc:2369] T c3828df18d3a4774a0053986b0f1baae P fdfbafe3ba214448b087cf163e74c75a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 05:01:26.272637 14860 tablet.cc:2369] T c3828df18d3a4774a0053986b0f1baae P c60634e3d40f4fdb9b890b2d27481143: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 05:01:26.275861 15130 tablet.cc:2369] T c3828df18d3a4774a0053986b0f1baae P 42765acdc69b496cba88f5821ca315cc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 05:01:26.361263 15144 raft_consensus.cc:482] T c3828df18d3a4774a0053986b0f1baae P 42765acdc69b496cba88f5821ca315cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 05:01:26.361447 15144 raft_consensus.cc:504] T c3828df18d3a4774a0053986b0f1baae P 42765acdc69b496cba88f5821ca315cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42765acdc69b496cba88f5821ca315cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43015 } } peers { permanent_uuid: "fdfbafe3ba214448b087cf163e74c75a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37425 } } peers { permanent_uuid: "c60634e3d40f4fdb9b890b2d27481143" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38433 } }
I20260924 05:01:26.361879 15144 leader_election.cc:288] T c3828df18d3a4774a0053986b0f1baae P 42765acdc69b496cba88f5821ca315cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fdfbafe3ba214448b087cf163e74c75a (127.0.98.2:37425), c60634e3d40f4fdb9b890b2d27481143 (127.0.98.1:38433)
I20260924 05:01:26.364846 15143 raft_consensus.cc:482] T c3828df18d3a4774a0053986b0f1baae P fdfbafe3ba214448b087cf163e74c75a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 05:01:26.364960 15143 raft_consensus.cc:504] T c3828df18d3a4774a0053986b0f1baae P fdfbafe3ba214448b087cf163e74c75a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42765acdc69b496cba88f5821ca315cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43015 } } peers { permanent_uuid: "fdfbafe3ba214448b087cf163e74c75a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37425 } } peers { permanent_uuid: "c60634e3d40f4fdb9b890b2d27481143" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38433 } }
I20260924 05:01:26.365293 15143 leader_election.cc:288] T c3828df18d3a4774a0053986b0f1baae P fdfbafe3ba214448b087cf163e74c75a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42765acdc69b496cba88f5821ca315cc (127.0.98.3:43015), c60634e3d40f4fdb9b890b2d27481143 (127.0.98.1:38433)
I20260924 05:01:26.365638 15146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:26.362080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38433 (local address 127.0.98.3:52091)
0924 05:01:26.362265 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:26.362274 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:01:26.362337 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:01:26.362489 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:01:26.362489 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:01:26.362491 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:01:26.362537 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:01:26.362538 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:01:26.363373 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:01:26.363374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:01:26.364624 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:01:26.364627 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:01:26.365356 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:01:26.365357 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:01:26.365441 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:26.365446 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 05:01:26.365495 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 05:01:26.365546 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":11,"thread_start_us":100,"threads_started":1}
I20260924 05:01:26.365725 15137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:26.362155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41503 (local address 127.0.98.2:37425)
0924 05:01:26.362213 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 05:01:26.362215 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:01:26.362225 (+ 10us) server_negotiation.cc:407] Connection header received
0924 05:01:26.362255 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:01:26.362255 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:01:26.362268 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:01:26.362331 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:01:26.362489 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:26.363248 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:26.364736 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:26.365230 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:26.365301 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:26.365311 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 05:01:26.365631 (+ 320us) server_negotiation.cc:299] Negotiation successful
0924 05:01:26.365670 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 05:01:26.365857 15145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:26.362125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52091 (local address 127.0.98.1:38433)
0924 05:01:26.362310 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 05:01:26.362313 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:01:26.362322 (+ 9us) server_negotiation.cc:407] Connection header received
0924 05:01:26.362406 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:01:26.362407 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:01:26.362417 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:01:26.362479 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:01:26.362590 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:26.363294 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:26.364736 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:26.365266 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:26.365355 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:26.365370 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0924 05:01:26.365723 (+ 353us) server_negotiation.cc:299] Negotiation successful
0924 05:01:26.365778 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":60,"threads_started":1}
I20260924 05:01:26.366216 15131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:26.362062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37425 (local address 127.0.98.3:41503)
0924 05:01:26.362124 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:26.362135 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 05:01:26.362179 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:01:26.362348 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:01:26.362350 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:01:26.362351 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:01:26.362421 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:01:26.362423 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:01:26.363331 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:01:26.363332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:01:26.364624 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:01:26.364627 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:01:26.365311 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:01:26.365312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:01:26.365444 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:26.365446 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 05:01:26.365495 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 05:01:26.365546 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":79}
I20260924 05:01:26.366513 14951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3828df18d3a4774a0053986b0f1baae" candidate_uuid: "42765acdc69b496cba88f5821ca315cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfbafe3ba214448b087cf163e74c75a" is_pre_election: true
I20260924 05:01:26.366605 14951 raft_consensus.cc:2454] T c3828df18d3a4774a0053986b0f1baae P fdfbafe3ba214448b087cf163e74c75a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42765acdc69b496cba88f5821ca315cc in term 0.
I20260924 05:01:26.366793 15017 leader_election.cc:302] T c3828df18d3a4774a0053986b0f1baae P 42765acdc69b496cba88f5821ca315cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42765acdc69b496cba88f5821ca315cc, fdfbafe3ba214448b087cf163e74c75a; no voters:
I20260924 05:01:26.366915 15144 raft_consensus.cc:2781] T c3828df18d3a4774a0053986b0f1baae P 42765acdc69b496cba88f5821ca315cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 05:01:26.366962 15144 raft_consensus.cc:482] T c3828df18d3a4774a0053986b0f1baae P 42765acdc69b496cba88f5821ca315cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 05:01:26.366988 15144 raft_consensus.cc:3037] T c3828df18d3a4774a0053986b0f1baae P 42765acdc69b496cba88f5821ca315cc [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:01:26.367641 15144 raft_consensus.cc:504] T c3828df18d3a4774a0053986b0f1baae P 42765acdc69b496cba88f5821ca315cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42765acdc69b496cba88f5821ca315cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43015 } } peers { permanent_uuid: "fdfbafe3ba214448b087cf163e74c75a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37425 } } peers { permanent_uuid: "c60634e3d40f4fdb9b890b2d27481143" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38433 } }
I20260924 05:01:26.367744 15144 leader_election.cc:288] T c3828df18d3a4774a0053986b0f1baae P 42765acdc69b496cba88f5821ca315cc [CANDIDATE]: Term 1 election: Requested vote from peers fdfbafe3ba214448b087cf163e74c75a (127.0.98.2:37425), c60634e3d40f4fdb9b890b2d27481143 (127.0.98.1:38433)
I20260924 05:01:26.367913 14951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3828df18d3a4774a0053986b0f1baae" candidate_uuid: "42765acdc69b496cba88f5821ca315cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfbafe3ba214448b087cf163e74c75a"
I20260924 05:01:26.367985 14951 raft_consensus.cc:3037] T c3828df18d3a4774a0053986b0f1baae P fdfbafe3ba214448b087cf163e74c75a [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:01:26.368095 14813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3828df18d3a4774a0053986b0f1baae" candidate_uuid: "42765acdc69b496cba88f5821ca315cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c60634e3d40f4fdb9b890b2d27481143" is_pre_election: true
I20260924 05:01:26.368224 14813 raft_consensus.cc:2454] T c3828df18d3a4774a0053986b0f1baae P c60634e3d40f4fdb9b890b2d27481143 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42765acdc69b496cba88f5821ca315cc in term 0.
I20260924 05:01:26.368325 14811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3828df18d3a4774a0053986b0f1baae" candidate_uuid: "42765acdc69b496cba88f5821ca315cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c60634e3d40f4fdb9b890b2d27481143"
I20260924 05:01:26.368374 14811 raft_consensus.cc:3037] T c3828df18d3a4774a0053986b0f1baae P c60634e3d40f4fdb9b890b2d27481143 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:01:26.368793 14999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:26.365495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43015 (local address 127.0.98.2:49135)
0924 05:01:26.365545 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:26.365554 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:01:26.365596 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:01:26.365815 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:01:26.365816 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:01:26.365818 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:01:26.365869 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:01:26.365871 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:01:26.366698 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:01:26.366699 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:01:26.367853 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:01:26.367855 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:01:26.368562 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:01:26.368563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:01:26.368634 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:26.368637 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:01:26.368686 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 05:01:26.368732 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260924 05:01:26.368804 14951 raft_consensus.cc:2454] T c3828df18d3a4774a0053986b0f1baae P fdfbafe3ba214448b087cf163e74c75a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42765acdc69b496cba88f5821ca315cc in term 1.
I20260924 05:01:26.369025 15148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:26.365772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38563 (local address 127.0.98.1:38433)
0924 05:01:26.365876 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0924 05:01:26.365877 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:01:26.365990 (+ 113us) server_negotiation.cc:407] Connection header received
0924 05:01:26.366039 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:01:26.366040 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:01:26.366048 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:01:26.366112 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:01:26.366267 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:26.366921 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:26.368123 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:26.368668 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:26.368708 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:26.368717 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 05:01:26.368923 (+ 206us) server_negotiation.cc:299] Negotiation successful
0924 05:01:26.368967 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":58,"threads_started":1}
I20260924 05:01:26.369091 15136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:26.365662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49135 (local address 127.0.98.3:43015)
0924 05:01:26.365702 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 05:01:26.365704 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:01:26.365710 (+ 6us) server_negotiation.cc:407] Connection header received
0924 05:01:26.365728 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:01:26.365729 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:01:26.365737 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:01:26.365778 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:01:26.365939 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:26.366618 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:26.367955 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:26.368471 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:26.368525 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:26.368534 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 05:01:26.369008 (+ 474us) server_negotiation.cc:299] Negotiation successful
0924 05:01:26.369043 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 05:01:26.369158 14811 raft_consensus.cc:2454] T c3828df18d3a4774a0053986b0f1baae P c60634e3d40f4fdb9b890b2d27481143 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42765acdc69b496cba88f5821ca315cc in term 1.
I20260924 05:01:26.369180 15147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:26.365594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38433 (local address 127.0.98.2:38563)
0924 05:01:26.365934 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:26.365942 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 05:01:26.365992 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:01:26.366126 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:01:26.366127 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:01:26.366128 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:01:26.366188 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:01:26.366189 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:01:26.367017 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:01:26.367018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:01:26.368042 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:01:26.368044 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:01:26.368770 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:01:26.368771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:01:26.368844 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:26.368847 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:01:26.369077 (+ 230us) client_negotiation.cc:240] Negotiation successful
0924 05:01:26.369111 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":92,"threads_started":1}
I20260924 05:01:26.369095 15017 leader_election.cc:302] T c3828df18d3a4774a0053986b0f1baae P 42765acdc69b496cba88f5821ca315cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42765acdc69b496cba88f5821ca315cc, fdfbafe3ba214448b087cf163e74c75a; no voters:
I20260924 05:01:26.369311 15083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3828df18d3a4774a0053986b0f1baae" candidate_uuid: "fdfbafe3ba214448b087cf163e74c75a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42765acdc69b496cba88f5821ca315cc" is_pre_election: true
I20260924 05:01:26.369381 15144 raft_consensus.cc:2781] T c3828df18d3a4774a0053986b0f1baae P 42765acdc69b496cba88f5821ca315cc [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:01:26.369542 15144 raft_consensus.cc:686] T c3828df18d3a4774a0053986b0f1baae P 42765acdc69b496cba88f5821ca315cc [term 1 LEADER]: Becoming Leader. State: Replica: 42765acdc69b496cba88f5821ca315cc, State: Running, Role: LEADER
I20260924 05:01:26.369611 14811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3828df18d3a4774a0053986b0f1baae" candidate_uuid: "fdfbafe3ba214448b087cf163e74c75a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c60634e3d40f4fdb9b890b2d27481143" is_pre_election: true
I20260924 05:01:26.369684 14811 raft_consensus.cc:2379] T c3828df18d3a4774a0053986b0f1baae P c60634e3d40f4fdb9b890b2d27481143 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fdfbafe3ba214448b087cf163e74c75a in current term 1: Already voted for candidate 42765acdc69b496cba88f5821ca315cc in this term.
I20260924 05:01:26.369630 15144 consensus_queue.cc:237] T c3828df18d3a4774a0053986b0f1baae P 42765acdc69b496cba88f5821ca315cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42765acdc69b496cba88f5821ca315cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43015 } } peers { permanent_uuid: "fdfbafe3ba214448b087cf163e74c75a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37425 } } peers { permanent_uuid: "c60634e3d40f4fdb9b890b2d27481143" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38433 } }
I20260924 05:01:26.370323 14886 leader_election.cc:302] T c3828df18d3a4774a0053986b0f1baae P fdfbafe3ba214448b087cf163e74c75a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfbafe3ba214448b087cf163e74c75a; no voters: 42765acdc69b496cba88f5821ca315cc, c60634e3d40f4fdb9b890b2d27481143
I20260924 05:01:26.370433 15143 raft_consensus.cc:2726] T c3828df18d3a4774a0053986b0f1baae P fdfbafe3ba214448b087cf163e74c75a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 05:01:26.370391 14671 catalog_manager.cc:5495] T c3828df18d3a4774a0053986b0f1baae P 42765acdc69b496cba88f5821ca315cc reported cstate change: term changed from 0 to 1, leader changed from <none> to 42765acdc69b496cba88f5821ca315cc (127.0.98.3). New cstate: current_term: 1 leader_uuid: "42765acdc69b496cba88f5821ca315cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42765acdc69b496cba88f5821ca315cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43015 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdfbafe3ba214448b087cf163e74c75a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c60634e3d40f4fdb9b890b2d27481143" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38433 } health_report { overall_health: UNKNOWN } } }
I20260924 05:01:26.406739 15136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:26.404644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42798 (local address 127.0.98.3:43015)
0924 05:01:26.404690 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 05:01:26.404692 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:01:26.404699 (+ 7us) server_negotiation.cc:407] Connection header received
0924 05:01:26.404751 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:01:26.404752 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:01:26.404768 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:01:26.404825 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:01:26.404911 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:26.405388 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:26.406110 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:26.406337 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:26.406391 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:26.406478 (+ 87us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:01:26.406575 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:01:26.406625 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 05:01:26.406641 (+ 16us) server_negotiation.cc:299] Negotiation successful
0924 05:01:26.406656 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 05:01:26.408494 14811 raft_consensus.cc:1261] T c3828df18d3a4774a0053986b0f1baae P c60634e3d40f4fdb9b890b2d27481143 [term 1 FOLLOWER]: Refusing update from remote peer 42765acdc69b496cba88f5821ca315cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:01:26.408509 14951 raft_consensus.cc:1261] T c3828df18d3a4774a0053986b0f1baae P fdfbafe3ba214448b087cf163e74c75a [term 1 FOLLOWER]: Refusing update from remote peer 42765acdc69b496cba88f5821ca315cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:01:26.408761 15144 consensus_queue.cc:1035] T c3828df18d3a4774a0053986b0f1baae P 42765acdc69b496cba88f5821ca315cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c60634e3d40f4fdb9b890b2d27481143" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:01:26.408891 15149 consensus_queue.cc:1035] T c3828df18d3a4774a0053986b0f1baae P 42765acdc69b496cba88f5821ca315cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfbafe3ba214448b087cf163e74c75a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:01:26.411510 15153 mvcc.cc:205] Tried to move back new op lower bound from 7332766049926504448 to 7332766049770438656. Current Snapshot: MvccSnapshot[applied={T|T < 7332766049926504448}]
I20260924 05:01:26.411626 15152 mvcc.cc:205] Tried to move back new op lower bound from 7332766049926504448 to 7332766049770438656. Current Snapshot: MvccSnapshot[applied={T|T < 7332766049926504448 or (T in {7332766049926504448})}]
I20260924 05:01:26.413465 14670 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:43544:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:01:26.413551 14670 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:43544:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:01:26.414188 14670 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7725fcfb576b446cafbc9944edacc6b4: Sending DeleteTablet for 3 replicas of tablet c3828df18d3a4774a0053986b0f1baae
I20260924 05:01:26.414485 14931 tablet_service.cc:1514] Processing DeleteTablet for tablet c3828df18d3a4774a0053986b0f1baae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:01:26 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50994
I20260924 05:01:26.414562 14793 tablet_service.cc:1514] Processing DeleteTablet for tablet c3828df18d3a4774a0053986b0f1baae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:01:26 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50174
I20260924 05:01:26.414558 15063 tablet_service.cc:1514] Processing DeleteTablet for tablet c3828df18d3a4774a0053986b0f1baae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:01:26 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42794
I20260924 05:01:26.414754 15163 tablet_replica.cc:330] T c3828df18d3a4774a0053986b0f1baae P 42765acdc69b496cba88f5821ca315cc: stopping tablet replica
I20260924 05:01:26.414793 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 14709
I20260924 05:01:26.414841 15163 raft_consensus.cc:2229] T c3828df18d3a4774a0053986b0f1baae P 42765acdc69b496cba88f5821ca315cc [term 1 LEADER]: Raft consensus shutting down.
I20260924 05:01:26.414932 15163 raft_consensus.cc:2258] T c3828df18d3a4774a0053986b0f1baae P 42765acdc69b496cba88f5821ca315cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:01:26.415341 15163 ts_tablet_manager.cc:1905] T c3828df18d3a4774a0053986b0f1baae P 42765acdc69b496cba88f5821ca315cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:01:26.417276 15163 ts_tablet_manager.cc:1918] T c3828df18d3a4774a0053986b0f1baae P 42765acdc69b496cba88f5821ca315cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 05:01:26.417336 15163 log.cc:1196] T c3828df18d3a4774a0053986b0f1baae P 42765acdc69b496cba88f5821ca315cc: Deleting WAL directory at /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-2/wal/wals/c3828df18d3a4774a0053986b0f1baae
I20260924 05:01:26.417619 15163 ts_tablet_manager.cc:1939] T c3828df18d3a4774a0053986b0f1baae P 42765acdc69b496cba88f5821ca315cc: Deleting consensus metadata
I20260924 05:01:26.417963 14657 catalog_manager.cc:4841] TS 42765acdc69b496cba88f5821ca315cc (127.0.98.3:43015): tablet c3828df18d3a4774a0053986b0f1baae (table test-table [id=e508c6471c2047cdab1cb00e16334cb5]) successfully deleted
I20260924 05:01:26.421543 15161 tablet_replica.cc:330] T c3828df18d3a4774a0053986b0f1baae P fdfbafe3ba214448b087cf163e74c75a: stopping tablet replica
I20260924 05:01:26.421653 15161 raft_consensus.cc:2229] T c3828df18d3a4774a0053986b0f1baae P fdfbafe3ba214448b087cf163e74c75a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:01:26.421775 15161 raft_consensus.cc:2258] T c3828df18d3a4774a0053986b0f1baae P fdfbafe3ba214448b087cf163e74c75a [term 1 FOLLOWER]: Raft consensus is shut down!
W20260924 05:01:26.421934 14657 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38433 (error 108)
I20260924 05:01:26.421996 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 14864
I20260924 05:01:26.422883 14867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:26.422767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38433 (local address 127.0.0.1:50180)
0924 05:01:26.422807 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:26.422838 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260924 05:01:26.422977 14657 catalog_manager.cc:4584] TS c60634e3d40f4fdb9b890b2d27481143 (127.0.98.1:38433): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c3828df18d3a4774a0053986b0f1baae: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38433: connect: Connection refused (error 111)
I20260924 05:01:26.426440 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 15000
I20260924 05:01:26.426517 14867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:26.426408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37425 (local address 127.0.0.1:51006)
0924 05:01:26.426462 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:26.426485 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:37425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260924 05:01:26.430328 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 14638
I20260924 05:01:26.434787 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43939
--webserver_interface=127.0.98.62
--webserver_port=34763
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43939
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:01:26.513221 15165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:01:26.513442 15165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:01:26.513475 15165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:01:26.514621 15165 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:01:26.514660 15165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:01:26.514678 15165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:01:26.514694 15165 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:01:26.514709 15165 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:01:26.516160 15165 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43939
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43939
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=34763
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15165
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:01:26.516398 15165 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:01:26.516597 15165 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 05:01:26.521052 15165 server_base.cc:1034] running on GCE node
W20260924 05:01:26.521025 15173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:01:26.521025 15171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:01:26.521019 15170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:01:26.521550 15165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:01:26.521824 15165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:01:26.522969 15165 hybrid_clock.cc:648] HybridClock initialized: now 1790226086522954 us; error 29 us; skew 500 ppm
Sep 24 05:01:26 dist-test-slave-70st krb5kdc[14094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226086, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:01:26.525246 15165 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 05:01:26.525682 15165 webserver.cc:460] Webserver started at http://127.0.98.62:34763/ using document root <none> and password file <none>
I20260924 05:01:26.525870 15165 fs_manager.cc:359] Metadata directory not provided
I20260924 05:01:26.525918 15165 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:01:26.527194 15165 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 05:01:26.527801 15180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:01:26.527967 15165 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:01:26.528054 15165 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/master-0/wal
uuid: "7725fcfb576b446cafbc9944edacc6b4"
format_stamp: "Formatted at 2026-09-24 05:01:24 on dist-test-slave-70st"
server_key: "d37b7160a44d8360610ee8ce5d87c5d9"
server_key_iv: "34c8bfdc26c228542a257500188c6491"
server_key_version: "encryptionkey@0"
I20260924 05:01:26.528388 15165 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:01:26.549757 15165 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:01:26.551925 15165 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:01:26.552073 15165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:01:26.555737 15165 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43939
I20260924 05:01:26.555773 15232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43939 every 8 connection(s)
I20260924 05:01:26.556134 15165 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:01:26.557603 15233 sys_catalog.cc:263] Verifying existing consensus state
I20260924 05:01:26.558229 15233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7725fcfb576b446cafbc9944edacc6b4: Bootstrap starting.
I20260924 05:01:26.559083 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 15165
I20260924 05:01:26.559317 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:38433
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:43939
--webserver_port=39289
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 05:01:26.567554 15233 log.cc:824] T 00000000000000000000000000000000 P 7725fcfb576b446cafbc9944edacc6b4: Log is configured to *not* fsync() on all Append() calls
I20260924 05:01:26.569878 15233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7725fcfb576b446cafbc9944edacc6b4: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 05:01:26.570163 15233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7725fcfb576b446cafbc9944edacc6b4: Bootstrap complete.
I20260924 05:01:26.572201 15233 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7725fcfb576b446cafbc9944edacc6b4 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7725fcfb576b446cafbc9944edacc6b4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43939 } }
I20260924 05:01:26.572420 15233 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7725fcfb576b446cafbc9944edacc6b4 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7725fcfb576b446cafbc9944edacc6b4, State: Initialized, Role: FOLLOWER
I20260924 05:01:26.572572 15233 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7725fcfb576b446cafbc9944edacc6b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7725fcfb576b446cafbc9944edacc6b4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43939 } }
I20260924 05:01:26.572638 15233 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7725fcfb576b446cafbc9944edacc6b4 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:01:26.572665 15233 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7725fcfb576b446cafbc9944edacc6b4 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:01:26.572710 15233 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7725fcfb576b446cafbc9944edacc6b4 [term 2 FOLLOWER]: Advancing to term 3
I20260924 05:01:26.573743 15233 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7725fcfb576b446cafbc9944edacc6b4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7725fcfb576b446cafbc9944edacc6b4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43939 } }
I20260924 05:01:26.573833 15233 leader_election.cc:302] T 00000000000000000000000000000000 P 7725fcfb576b446cafbc9944edacc6b4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7725fcfb576b446cafbc9944edacc6b4; no voters:
I20260924 05:01:26.574002 15233 leader_election.cc:288] T 00000000000000000000000000000000 P 7725fcfb576b446cafbc9944edacc6b4 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 05:01:26.574057 15237 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7725fcfb576b446cafbc9944edacc6b4 [term 3 FOLLOWER]: Leader election won for term 3
I20260924 05:01:26.574195 15237 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7725fcfb576b446cafbc9944edacc6b4 [term 3 LEADER]: Becoming Leader. State: Replica: 7725fcfb576b446cafbc9944edacc6b4, State: Running, Role: LEADER
I20260924 05:01:26.574262 15233 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7725fcfb576b446cafbc9944edacc6b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:01:26.574395 15237 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7725fcfb576b446cafbc9944edacc6b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7725fcfb576b446cafbc9944edacc6b4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43939 } }
I20260924 05:01:26.574705 15238 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7725fcfb576b446cafbc9944edacc6b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "7725fcfb576b446cafbc9944edacc6b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7725fcfb576b446cafbc9944edacc6b4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43939 } } }
I20260924 05:01:26.574779 15238 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7725fcfb576b446cafbc9944edacc6b4 [sys.catalog]: This master's current role is: LEADER
I20260924 05:01:26.574987 15239 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7725fcfb576b446cafbc9944edacc6b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7725fcfb576b446cafbc9944edacc6b4. Latest consensus state: current_term: 3 leader_uuid: "7725fcfb576b446cafbc9944edacc6b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7725fcfb576b446cafbc9944edacc6b4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43939 } } }
I20260924 05:01:26.575106 15239 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7725fcfb576b446cafbc9944edacc6b4 [sys.catalog]: This master's current role is: LEADER
I20260924 05:01:26.575223 15245 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:01:26.576050 15245 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:01:26.576361 15245 catalog_manager.cc:1259] Loaded cluster ID: 46f0a1bdd2da4de58d54f3cb3ba37607
I20260924 05:01:26.576400 15245 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260924 05:01:26.576603 15245 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f8fcd65f9ad google::LogMessage::Fail() at ??:0
@ 0x7f8fcd66288a google::LogMessage::SendToLog() at ??:0
@ 0x7f8fcd65f4de google::LogMessage::Flush() at ??:0
@ 0x7f8fcd660a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f8fd55800f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f8fd55898b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f8fcf52d0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f8fcf522954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f8fd03866db start_thread at ??:0
@ 0x7f8fcc15d61f clone at ??:0
W20260924 05:01:26.647363 15235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:01:26.647558 15235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:01:26.647590 15235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:01:26.648748 15235 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:01:26.648798 15235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:01:26.648834 15235 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 05:01:26.650302 15235 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:38433
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=39289
--tserver_master_addrs=127.0.98.62:43939
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15235
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:01:26.650533 15235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:01:26.650740 15235 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:01:26.655370 15272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:01:26.655416 15269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:01:26.655650 15235 server_base.cc:1034] running on GCE node
W20260924 05:01:26.655378 15270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:01:26.655889 15235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:01:26.656109 15235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:01:26.657280 15235 hybrid_clock.cc:648] HybridClock initialized: now 1790226086657259 us; error 37 us; skew 500 ppm
Sep 24 05:01:26 dist-test-slave-70st krb5kdc[14094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226086, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:01:26.659864 15235 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 05:01:26.660378 15235 webserver.cc:460] Webserver started at http://127.0.98.1:39289/ using document root <none> and password file <none>
I20260924 05:01:26.660660 15235 fs_manager.cc:359] Metadata directory not provided
I20260924 05:01:26.660730 15235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:01:26.662523 15235 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 05:01:26.663275 15279 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:01:26.663447 15235 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 05:01:26.663525 15235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-0/wal
uuid: "c60634e3d40f4fdb9b890b2d27481143"
format_stamp: "Formatted at 2026-09-24 05:01:24 on dist-test-slave-70st"
server_key: "57b7b43a095930ff567c4befa65feeb5"
server_key_iv: "049cc7180b3f42ed314190f68ff24118"
server_key_version: "encryptionkey@0"
I20260924 05:01:26.663884 15235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:01:26.675513 15235 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:01:26.677603 15235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:01:26.677727 15235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:01:26.677963 15235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:01:26.678427 15286 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 05:01:26.680488 15235 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 05:01:26.680533 15235 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260924 05:01:26.680567 15235 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 05:01:26.681159 15235 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 05:01:26.681191 15235 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260924 05:01:26.681298 15286 tablet_bootstrap.cc:492] T c3828df18d3a4774a0053986b0f1baae P c60634e3d40f4fdb9b890b2d27481143: Bootstrap starting.
I20260924 05:01:26.687965 15235 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38433
I20260924 05:01:26.688136 15393 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38433 every 8 connection(s)
I20260924 05:01:26.688392 15235 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-0/data/info.pb
I20260924 05:01:26.692052 15286 log.cc:824] T c3828df18d3a4774a0053986b0f1baae P c60634e3d40f4fdb9b890b2d27481143: Log is configured to *not* fsync() on all Append() calls
I20260924 05:01:26.694658 15286 tablet_bootstrap.cc:492] T c3828df18d3a4774a0053986b0f1baae P c60634e3d40f4fdb9b890b2d27481143: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 05:01:26.695007 15286 tablet_bootstrap.cc:492] T c3828df18d3a4774a0053986b0f1baae P c60634e3d40f4fdb9b890b2d27481143: Bootstrap complete.
I20260924 05:01:26.695957 15286 ts_tablet_manager.cc:1397] T c3828df18d3a4774a0053986b0f1baae P c60634e3d40f4fdb9b890b2d27481143: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.010s
I20260924 05:01:26.697118 15286 raft_consensus.cc:348] T c3828df18d3a4774a0053986b0f1baae P c60634e3d40f4fdb9b890b2d27481143 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42765acdc69b496cba88f5821ca315cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43015 } } peers { permanent_uuid: "fdfbafe3ba214448b087cf163e74c75a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37425 } } peers { permanent_uuid: "c60634e3d40f4fdb9b890b2d27481143" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38433 } }
I20260924 05:01:26.697500 15286 raft_consensus.cc:729] T c3828df18d3a4774a0053986b0f1baae P c60634e3d40f4fdb9b890b2d27481143 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c60634e3d40f4fdb9b890b2d27481143, State: Initialized, Role: FOLLOWER
I20260924 05:01:26.697651 15286 consensus_queue.cc:260] T c3828df18d3a4774a0053986b0f1baae P c60634e3d40f4fdb9b890b2d27481143 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42765acdc69b496cba88f5821ca315cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43015 } } peers { permanent_uuid: "fdfbafe3ba214448b087cf163e74c75a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37425 } } peers { permanent_uuid: "c60634e3d40f4fdb9b890b2d27481143" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38433 } }
I20260924 05:01:26.697965 15286 ts_tablet_manager.cc:1428] T c3828df18d3a4774a0053986b0f1baae P c60634e3d40f4fdb9b890b2d27481143: Time spent starting tablet: real 0.002s user 0.003s sys 0.001s
I20260924 05:01:26.698460 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 15235
I20260924 05:01:26.698717 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:37425
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:43939
--webserver_port=46749
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:01:26.816054 15399 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:01:26.816318 15399 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:01:26.816354 15399 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:01:26.817919 15399 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:01:26.817977 15399 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:01:26.818013 15399 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 05:01:26.820025 15399 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:37425
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=46749
--tserver_master_addrs=127.0.98.62:43939
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15399
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:01:26.820279 15399 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:01:26.820547 15399 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:01:26.825778 15404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:01:26.825827 15407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:01:26.825784 15405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:01:26.826053 15399 server_base.cc:1034] running on GCE node
I20260924 05:01:26.826225 15399 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:01:26.826471 15399 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:01:26.827646 15399 hybrid_clock.cc:648] HybridClock initialized: now 1790226086827630 us; error 30 us; skew 500 ppm
Sep 24 05:01:26 dist-test-slave-70st krb5kdc[14094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226086, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:01:26.830250 15399 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 05:01:26.830708 15399 webserver.cc:460] Webserver started at http://127.0.98.2:46749/ using document root <none> and password file <none>
I20260924 05:01:26.830947 15399 fs_manager.cc:359] Metadata directory not provided
I20260924 05:01:26.831001 15399 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:01:26.832412 15399 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 05:01:26.833159 15414 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:01:26.833364 15399 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 05:01:26.833485 15399 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-1/wal
uuid: "fdfbafe3ba214448b087cf163e74c75a"
format_stamp: "Formatted at 2026-09-24 05:01:24 on dist-test-slave-70st"
server_key: "d8df35ec213e6c68fead8d0c0c14d84c"
server_key_iv: "781a03fd7a040819393296fa0b07da22"
server_key_version: "encryptionkey@0"
I20260924 05:01:26.833868 15399 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20260924 05:01:26.839066 15396 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:01:26.689224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43939 (local address 127.0.98.1:48735)
0924 05:01:26.689396 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:26.689461 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0924 05:01:26.689925 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:01:26.838872 (+148947us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43939: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":78,"threads_started":1}
W20260924 05:01:26.839327 15394 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43939 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:43939: Client connection negotiation failed: client connection to 127.0.98.62:43939: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 05:01:26.839639 15396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:26.839529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43939 (local address 127.0.98.1:36453)
0924 05:01:26.839576 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:26.839613 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260924 05:01:26.839974 15396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:26.839893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43939 (local address 127.0.98.1:59017)
0924 05:01:26.839934 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:26.839957 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260924 05:01:26.840075 15394 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 05:01:26.840873 15399 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:01:26.842935 15399 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:01:26.843056 15399 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:01:26.843259 15399 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:01:26.843667 15421 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 05:01:26.845041 15399 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 05:01:26.845080 15399 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260924 05:01:26.845111 15399 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 05:01:26.845764 15399 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 05:01:26.845796 15399 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260924 05:01:26.845873 15421 tablet_bootstrap.cc:492] T c3828df18d3a4774a0053986b0f1baae P fdfbafe3ba214448b087cf163e74c75a: Bootstrap starting.
I20260924 05:01:26.850941 15399 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37425
I20260924 05:01:26.851114 15528 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37425 every 8 connection(s)
I20260924 05:01:26.851332 15399 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-1/data/info.pb
I20260924 05:01:26.852551 15531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:26.851960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43939 (local address 127.0.98.2:43105)
0924 05:01:26.852253 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:26.852345 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":92,"threads_started":1}
W20260924 05:01:26.852747 15529 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43939 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:43939: Client connection negotiation failed: client connection to 127.0.98.62:43939: connect: Connection refused (error 111)
I20260924 05:01:26.853032 15531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:26.852928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43939 (local address 127.0.98.2:58769)
0924 05:01:26.852977 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:26.853011 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260924 05:01:26.853291 15531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:26.853210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43939 (local address 127.0.98.2:48527)
0924 05:01:26.853247 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:26.853272 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260924 05:01:26.853399 15529 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 05:01:26.854293 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 15399
I20260924 05:01:26.854542 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:43015
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:43939
--webserver_port=35705
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 05:01:26.855697 15421 log.cc:824] T c3828df18d3a4774a0053986b0f1baae P fdfbafe3ba214448b087cf163e74c75a: Log is configured to *not* fsync() on all Append() calls
I20260924 05:01:26.857900 15421 tablet_bootstrap.cc:492] T c3828df18d3a4774a0053986b0f1baae P fdfbafe3ba214448b087cf163e74c75a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 05:01:26.858130 15421 tablet_bootstrap.cc:492] T c3828df18d3a4774a0053986b0f1baae P fdfbafe3ba214448b087cf163e74c75a: Bootstrap complete.
I20260924 05:01:26.858909 15421 ts_tablet_manager.cc:1397] T c3828df18d3a4774a0053986b0f1baae P fdfbafe3ba214448b087cf163e74c75a: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.010s
I20260924 05:01:26.860034 15421 raft_consensus.cc:348] T c3828df18d3a4774a0053986b0f1baae P fdfbafe3ba214448b087cf163e74c75a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42765acdc69b496cba88f5821ca315cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43015 } } peers { permanent_uuid: "fdfbafe3ba214448b087cf163e74c75a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37425 } } peers { permanent_uuid: "c60634e3d40f4fdb9b890b2d27481143" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38433 } }
I20260924 05:01:26.860283 15421 raft_consensus.cc:729] T c3828df18d3a4774a0053986b0f1baae P fdfbafe3ba214448b087cf163e74c75a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdfbafe3ba214448b087cf163e74c75a, State: Initialized, Role: FOLLOWER
I20260924 05:01:26.860392 15421 consensus_queue.cc:260] T c3828df18d3a4774a0053986b0f1baae P fdfbafe3ba214448b087cf163e74c75a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42765acdc69b496cba88f5821ca315cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43015 } } peers { permanent_uuid: "fdfbafe3ba214448b087cf163e74c75a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37425 } } peers { permanent_uuid: "c60634e3d40f4fdb9b890b2d27481143" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38433 } }
I20260924 05:01:26.860630 15421 ts_tablet_manager.cc:1428] T c3828df18d3a4774a0053986b0f1baae P fdfbafe3ba214448b087cf163e74c75a: Time spent starting tablet: real 0.002s user 0.003s sys 0.001s
I20260924 05:01:26.860852 15531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:26.860766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43939 (local address 127.0.98.2:40407)
0924 05:01:26.860807 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:26.860834 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260924 05:01:26.861104 15531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:26.861052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43939 (local address 127.0.98.2:46937)
0924 05:01:26.861075 (+ 23us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:26.861092 (+ 17us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260924 05:01:26.933648 15532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:01:26.933851 15532 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:01:26.933876 15532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:01:26.934991 15532 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:01:26.935026 15532 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:01:26.935050 15532 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 05:01:26.936457 15532 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:43015
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=35705
--tserver_master_addrs=127.0.98.62:43939
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15532
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:01:26.936645 15532 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:01:26.936842 15532 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:01:26.941509 15540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:01:26.941499 15539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:01:26.941617 15532 server_base.cc:1034] running on GCE node
W20260924 05:01:26.941504 15542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:01:26.941908 15532 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:01:26.942121 15532 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:01:26.943275 15532 hybrid_clock.cc:648] HybridClock initialized: now 1790226086943262 us; error 29 us; skew 500 ppm
Sep 24 05:01:26 dist-test-slave-70st krb5kdc[14094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226086, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:01:26.945578 15532 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 05:01:26.945932 15532 webserver.cc:460] Webserver started at http://127.0.98.3:35705/ using document root <none> and password file <none>
I20260924 05:01:26.946094 15532 fs_manager.cc:359] Metadata directory not provided
I20260924 05:01:26.946130 15532 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:01:26.947356 15532 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 05:01:26.947872 15549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:01:26.948019 15532 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:01:26.948098 15532 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-2/wal
uuid: "42765acdc69b496cba88f5821ca315cc"
format_stamp: "Formatted at 2026-09-24 05:01:25 on dist-test-slave-70st"
server_key: "7bb3e281fe8192469659a4e6eb9f1c1c"
server_key_iv: "6687bc5781024942762f1338001d66cc"
server_key_version: "encryptionkey@0"
I20260924 05:01:26.948407 15532 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:01:26.964396 15532 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:01:26.966475 15532 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:01:26.966590 15532 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:01:26.966805 15532 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:01:26.967068 15532 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:01:26.967098 15532 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:01:26.967118 15532 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:01:26.967135 15532 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:01:26.972390 15532 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43015
I20260924 05:01:26.972461 15662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43015 every 8 connection(s)
I20260924 05:01:26.972759 15532 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790225929848825-392-0/minicluster-data/ts-2/data/info.pb
I20260924 05:01:26.973809 15665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:26.973407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43939 (local address 127.0.98.3:45797)
0924 05:01:26.973588 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:26.973659 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":76,"threads_started":1}
W20260924 05:01:26.974184 15663 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43939 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:43939: Client connection negotiation failed: client connection to 127.0.98.62:43939: connect: Connection refused (error 111)
I20260924 05:01:26.974483 15665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:26.974391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43939 (local address 127.0.98.3:48881)
0924 05:01:26.974436 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:26.974463 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260924 05:01:26.974701 15665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:26.974644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43939 (local address 127.0.98.3:43143)
0924 05:01:26.974672 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:26.974689 (+ 17us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260924 05:01:26.974777 15663 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 05:01:26.979439 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 15532
I20260924 05:01:27.840647 15667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:27.840419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43939 (local address 127.0.98.1:36795)
0924 05:01:27.840542 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:27.840583 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":62,"threads_started":1}
I20260924 05:01:27.861689 15668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:27.861486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43939 (local address 127.0.98.2:51447)
0924 05:01:27.861599 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:27.861634 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":54,"threads_started":1}
I20260924 05:01:27.975337 15669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:27.975099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43939 (local address 127.0.98.3:38337)
0924 05:01:27.975233 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:27.975277 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":67,"threads_started":1}
I20260924 05:01:28.424973 15670 raft_consensus.cc:482] T c3828df18d3a4774a0053986b0f1baae P c60634e3d40f4fdb9b890b2d27481143 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 05:01:28.425096 15670 raft_consensus.cc:504] T c3828df18d3a4774a0053986b0f1baae P c60634e3d40f4fdb9b890b2d27481143 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42765acdc69b496cba88f5821ca315cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43015 } } peers { permanent_uuid: "fdfbafe3ba214448b087cf163e74c75a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37425 } } peers { permanent_uuid: "c60634e3d40f4fdb9b890b2d27481143" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38433 } }
I20260924 05:01:28.425457 15670 leader_election.cc:288] T c3828df18d3a4774a0053986b0f1baae P c60634e3d40f4fdb9b890b2d27481143 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42765acdc69b496cba88f5821ca315cc (127.0.98.3:43015), fdfbafe3ba214448b087cf163e74c75a (127.0.98.2:37425)
Sep 24 05:01:28 dist-test-slave-70st krb5kdc[14094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226086, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Sep 24 05:01:28 dist-test-slave-70st krb5kdc[14094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226086, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260924 05:01:28.435253 15672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:28.425602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37425 (local address 127.0.98.1:49629)
0924 05:01:28.425821 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:28.425830 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:01:28.425866 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:01:28.426247 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:01:28.426250 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:01:28.426641 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:01:28.426877 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:01:28.426884 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:01:28.428082 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:01:28.428083 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:01:28.428862 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:01:28.428864 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:01:28.429438 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:01:28.429439 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:01:28.429553 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:28.430119 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:01:28.430123 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:01:28.431812 (+ 1689us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:01:28.433564 (+ 1752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:01:28.433571 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:01:28.433573 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:01:28.433866 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:01:28.434130 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:01:28.434131 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:01:28.434131 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:01:28.434259 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:01:28.434683 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:01:28.434689 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:01:28.434836 (+ 147us) client_negotiation.cc:769] Sending connection context
0924 05:01:28.434947 (+ 111us) client_negotiation.cc:240] Negotiation successful
0924 05:01:28.435127 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"mutex_wait_us":64,"spinlock_wait_cycles":13184,"thread_start_us":88,"threads_started":1}
I20260924 05:01:28.435343 15671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:28.425602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43015 (local address 127.0.98.1:59687)
0924 05:01:28.425756 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:28.425768 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 05:01:28.425827 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:01:28.426708 (+ 881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:01:28.426708 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:01:28.426875 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:01:28.426936 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:01:28.426938 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:01:28.428014 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:01:28.428015 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:01:28.428861 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:01:28.428863 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:01:28.429448 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:01:28.429449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:01:28.429551 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:28.430067 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:01:28.430075 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:01:28.431571 (+ 1496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:01:28.433950 (+ 2379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:01:28.433951 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:01:28.433951 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:01:28.434147 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:01:28.434433 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:01:28.434433 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:01:28.434434 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:01:28.434488 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:01:28.434913 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:01:28.434914 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:01:28.434977 (+ 63us) client_negotiation.cc:769] Sending connection context
0924 05:01:28.435061 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 05:01:28.435202 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"mutex_wait_us":96,"thread_start_us":95,"threads_started":1}
I20260924 05:01:28.435560 15673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:28.425682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59687 (local address 127.0.98.3:43015)
0924 05:01:28.425944 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0924 05:01:28.425953 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 05:01:28.425977 (+ 24us) server_negotiation.cc:407] Connection header received
0924 05:01:28.426523 (+ 546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:01:28.426526 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:01:28.426574 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:01:28.426720 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:01:28.427030 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:28.427866 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:28.428945 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:28.429336 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:28.429462 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:28.431941 (+ 2479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:01:28.431945 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:01:28.431945 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:01:28.431957 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:01:28.433883 (+ 1926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:01:28.434259 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:01:28.434262 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:01:28.434262 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:01:28.434369 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:01:28.434545 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:01:28.434546 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:01:28.434546 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:01:28.434809 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:01:28.434899 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 05:01:28.435132 (+ 233us) server_negotiation.cc:299] Negotiation successful
0924 05:01:28.435293 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":62,"threads_started":1}
I20260924 05:01:28.435300 15674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:28.425763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49629 (local address 127.0.98.2:37425)
0924 05:01:28.425883 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0924 05:01:28.425893 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 05:01:28.425925 (+ 32us) server_negotiation.cc:407] Connection header received
0924 05:01:28.426026 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:01:28.426028 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:01:28.426067 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:01:28.426177 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:01:28.426978 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:28.427974 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:28.428963 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:28.429337 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:28.429457 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:28.431902 (+ 2445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:01:28.431906 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:01:28.431907 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:01:28.431916 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:01:28.433491 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:01:28.433949 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:01:28.433952 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:01:28.433952 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:01:28.434063 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:01:28.434325 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:01:28.434326 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:01:28.434326 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:01:28.434619 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:01:28.434676 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 05:01:28.435052 (+ 376us) server_negotiation.cc:299] Negotiation successful
0924 05:01:28.435196 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":46,"threads_started":1}
I20260924 05:01:28.436137 15483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3828df18d3a4774a0053986b0f1baae" candidate_uuid: "c60634e3d40f4fdb9b890b2d27481143" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "fdfbafe3ba214448b087cf163e74c75a" is_pre_election: true
I20260924 05:01:28.436291 15483 raft_consensus.cc:2454] T c3828df18d3a4774a0053986b0f1baae P fdfbafe3ba214448b087cf163e74c75a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c60634e3d40f4fdb9b890b2d27481143 in term 1.
W20260924 05:01:28.436378 15282 leader_election.cc:341] T c3828df18d3a4774a0053986b0f1baae P c60634e3d40f4fdb9b890b2d27481143 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 42765acdc69b496cba88f5821ca315cc (127.0.98.3:43015): Not found: Tablet not found: c3828df18d3a4774a0053986b0f1baae
I20260924 05:01:28.436530 15281 leader_election.cc:302] T c3828df18d3a4774a0053986b0f1baae P c60634e3d40f4fdb9b890b2d27481143 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c60634e3d40f4fdb9b890b2d27481143, fdfbafe3ba214448b087cf163e74c75a; no voters: 42765acdc69b496cba88f5821ca315cc
I20260924 05:01:28.436655 15670 raft_consensus.cc:2781] T c3828df18d3a4774a0053986b0f1baae P c60634e3d40f4fdb9b890b2d27481143 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 05:01:28.436726 15670 raft_consensus.cc:482] T c3828df18d3a4774a0053986b0f1baae P c60634e3d40f4fdb9b890b2d27481143 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 05:01:28.436757 15670 raft_consensus.cc:3037] T c3828df18d3a4774a0053986b0f1baae P c60634e3d40f4fdb9b890b2d27481143 [term 1 FOLLOWER]: Advancing to term 2
I20260924 05:01:28.437997 15670 raft_consensus.cc:504] T c3828df18d3a4774a0053986b0f1baae P c60634e3d40f4fdb9b890b2d27481143 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42765acdc69b496cba88f5821ca315cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43015 } } peers { permanent_uuid: "fdfbafe3ba214448b087cf163e74c75a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37425 } } peers { permanent_uuid: "c60634e3d40f4fdb9b890b2d27481143" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38433 } }
I20260924 05:01:28.438131 15670 leader_election.cc:288] T c3828df18d3a4774a0053986b0f1baae P c60634e3d40f4fdb9b890b2d27481143 [CANDIDATE]: Term 2 election: Requested vote from peers 42765acdc69b496cba88f5821ca315cc (127.0.98.3:43015), fdfbafe3ba214448b087cf163e74c75a (127.0.98.2:37425)
I20260924 05:01:28.438323 15483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3828df18d3a4774a0053986b0f1baae" candidate_uuid: "c60634e3d40f4fdb9b890b2d27481143" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "fdfbafe3ba214448b087cf163e74c75a"
I20260924 05:01:28.438431 15483 raft_consensus.cc:3037] T c3828df18d3a4774a0053986b0f1baae P fdfbafe3ba214448b087cf163e74c75a [term 1 FOLLOWER]: Advancing to term 2
W20260924 05:01:28.438519 15282 leader_election.cc:341] T c3828df18d3a4774a0053986b0f1baae P c60634e3d40f4fdb9b890b2d27481143 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 42765acdc69b496cba88f5821ca315cc (127.0.98.3:43015): Not found: Tablet not found: c3828df18d3a4774a0053986b0f1baae
I20260924 05:01:28.439514 15483 raft_consensus.cc:2454] T c3828df18d3a4774a0053986b0f1baae P fdfbafe3ba214448b087cf163e74c75a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c60634e3d40f4fdb9b890b2d27481143 in term 2.
I20260924 05:01:28.439682 15281 leader_election.cc:302] T c3828df18d3a4774a0053986b0f1baae P c60634e3d40f4fdb9b890b2d27481143 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c60634e3d40f4fdb9b890b2d27481143, fdfbafe3ba214448b087cf163e74c75a; no voters: 42765acdc69b496cba88f5821ca315cc
I20260924 05:01:28.439782 15670 raft_consensus.cc:2781] T c3828df18d3a4774a0053986b0f1baae P c60634e3d40f4fdb9b890b2d27481143 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 05:01:28.439913 15670 raft_consensus.cc:686] T c3828df18d3a4774a0053986b0f1baae P c60634e3d40f4fdb9b890b2d27481143 [term 2 LEADER]: Becoming Leader. State: Replica: c60634e3d40f4fdb9b890b2d27481143, State: Running, Role: LEADER
I20260924 05:01:28.439997 15670 consensus_queue.cc:237] T c3828df18d3a4774a0053986b0f1baae P c60634e3d40f4fdb9b890b2d27481143 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42765acdc69b496cba88f5821ca315cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43015 } } peers { permanent_uuid: "fdfbafe3ba214448b087cf163e74c75a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37425 } } peers { permanent_uuid: "c60634e3d40f4fdb9b890b2d27481143" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38433 } }
I20260924 05:01:28.440182 15671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:28.440089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43939 (local address 127.0.98.1:52513)
0924 05:01:28.440130 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:28.440157 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260924 05:01:28.862320 15677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:28.862096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43939 (local address 127.0.98.2:37441)
0924 05:01:28.862229 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:28.862267 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":57,"threads_started":1}
I20260924 05:01:28.902197 15483 raft_consensus.cc:1261] T c3828df18d3a4774a0053986b0f1baae P fdfbafe3ba214448b087cf163e74c75a [term 2 FOLLOWER]: Refusing update from remote peer c60634e3d40f4fdb9b890b2d27481143: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260924 05:01:28.902563 15670 consensus_queue.cc:1035] T c3828df18d3a4774a0053986b0f1baae P c60634e3d40f4fdb9b890b2d27481143 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfbafe3ba214448b087cf163e74c75a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260924 05:01:28.903383 15677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:28.903265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43939 (local address 127.0.98.2:50893)
0924 05:01:28.903326 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:28.903353 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260924 05:01:28.904244 15677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:28.904162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43939 (local address 127.0.98.2:58665)
0924 05:01:28.904193 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:28.904218 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260924 05:01:28.904315 15282 consensus_peers.cc:487] T c3828df18d3a4774a0053986b0f1baae P c60634e3d40f4fdb9b890b2d27481143 -> Peer 42765acdc69b496cba88f5821ca315cc (127.0.98.3:43015): Couldn't send request to peer 42765acdc69b496cba88f5821ca315cc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c3828df18d3a4774a0053986b0f1baae. This is attempt 1: this message will repeat every 5th retry.
I20260924 05:01:28.904403 15671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:28.904220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43939 (local address 127.0.98.1:54995)
0924 05:01:28.904347 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:28.904375 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260924 05:01:28.975966 15683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:28.975757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43939 (local address 127.0.98.3:45153)
0924 05:01:28.975878 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:28.975924 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":60,"threads_started":1}
I20260924 05:01:29.519798 15685 ts_tablet_manager.cc:927] T c3828df18d3a4774a0053986b0f1baae P 42765acdc69b496cba88f5821ca315cc: Initiating tablet copy from peer c60634e3d40f4fdb9b890b2d27481143 (127.0.98.1:38433)
I20260924 05:01:29.519994 15685 tablet_copy_client.cc:316] T c3828df18d3a4774a0053986b0f1baae P 42765acdc69b496cba88f5821ca315cc: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.1:38433
Sep 24 05:01:29 dist-test-slave-70st krb5kdc[14094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226086, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 05:01:29.528429 15687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:29.520277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38433 (local address 127.0.98.3:44437)
0924 05:01:29.520423 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:29.520444 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 05:01:29.520513 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:01:29.520688 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:01:29.520696 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:01:29.521072 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:01:29.521164 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:01:29.521167 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:01:29.522436 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:01:29.522437 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:01:29.523017 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:01:29.523019 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:01:29.523621 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:01:29.523622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:01:29.523700 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:29.524222 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:01:29.524227 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:01:29.525673 (+ 1446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:01:29.527378 (+ 1705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:01:29.527385 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:01:29.527388 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:01:29.527572 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:01:29.527786 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:01:29.527787 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:01:29.527787 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:01:29.527846 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:01:29.528142 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:01:29.528147 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:01:29.528191 (+ 44us) client_negotiation.cc:769] Sending connection context
0924 05:01:29.528250 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 05:01:29.528337 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":89,"threads_started":1}
I20260924 05:01:29.528435 15688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:29.520359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44437 (local address 127.0.98.1:38433)
0924 05:01:29.520481 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0924 05:01:29.520486 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 05:01:29.520498 (+ 12us) server_negotiation.cc:407] Connection header received
0924 05:01:29.520565 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:01:29.520568 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:01:29.520584 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:01:29.520653 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:01:29.521253 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:29.522363 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:29.523078 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:29.523541 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:29.523620 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:29.525740 (+ 2120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:01:29.525749 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:01:29.525750 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:01:29.525757 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:01:29.527314 (+ 1557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:01:29.527643 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:01:29.527646 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:01:29.527646 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:01:29.527704 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:01:29.527937 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:01:29.527938 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:01:29.527939 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:01:29.528089 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:01:29.528156 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 05:01:29.528280 (+ 124us) server_negotiation.cc:299] Negotiation successful
0924 05:01:29.528342 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84,"thread_start_us":55,"threads_started":1}
I20260924 05:01:29.528949 15368 tablet_copy_service.cc:140] P c60634e3d40f4fdb9b890b2d27481143: Received BeginTabletCopySession request for tablet c3828df18d3a4774a0053986b0f1baae from peer 42765acdc69b496cba88f5821ca315cc ({username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:44437)
I20260924 05:01:29.529011 15368 tablet_copy_service.cc:161] P c60634e3d40f4fdb9b890b2d27481143: Beginning new tablet copy session on tablet c3828df18d3a4774a0053986b0f1baae from peer 42765acdc69b496cba88f5821ca315cc at {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:44437: session id = 42765acdc69b496cba88f5821ca315cc-c3828df18d3a4774a0053986b0f1baae
I20260924 05:01:29.529511 15368 tablet_copy_source_session.cc:205] T c3828df18d3a4774a0053986b0f1baae P c60634e3d40f4fdb9b890b2d27481143: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 05:01:29.530391 15685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3828df18d3a4774a0053986b0f1baae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:01:29.532809 15685 tablet_copy_client.cc:791] T c3828df18d3a4774a0053986b0f1baae P 42765acdc69b496cba88f5821ca315cc: tablet copy: Starting download of 0 data blocks...
I20260924 05:01:29.532958 15685 tablet_copy_client.cc:655] T c3828df18d3a4774a0053986b0f1baae P 42765acdc69b496cba88f5821ca315cc: tablet copy: Starting download of 1 WAL segments...
I20260924 05:01:29.533648 15685 tablet_copy_client.cc:531] T c3828df18d3a4774a0053986b0f1baae P 42765acdc69b496cba88f5821ca315cc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 05:01:29.534256 15685 tablet_bootstrap.cc:492] T c3828df18d3a4774a0053986b0f1baae P 42765acdc69b496cba88f5821ca315cc: Bootstrap starting.
I20260924 05:01:29.535493 15685 log.cc:824] T c3828df18d3a4774a0053986b0f1baae P 42765acdc69b496cba88f5821ca315cc: Log is configured to *not* fsync() on all Append() calls
I20260924 05:01:29.537606 15685 tablet_bootstrap.cc:492] T c3828df18d3a4774a0053986b0f1baae P 42765acdc69b496cba88f5821ca315cc: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 05:01:29.537798 15685 tablet_bootstrap.cc:492] T c3828df18d3a4774a0053986b0f1baae P 42765acdc69b496cba88f5821ca315cc: Bootstrap complete.
I20260924 05:01:29.537915 15685 ts_tablet_manager.cc:1397] T c3828df18d3a4774a0053986b0f1baae P 42765acdc69b496cba88f5821ca315cc: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260924 05:01:29.538810 15685 raft_consensus.cc:348] T c3828df18d3a4774a0053986b0f1baae P 42765acdc69b496cba88f5821ca315cc [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42765acdc69b496cba88f5821ca315cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43015 } } peers { permanent_uuid: "fdfbafe3ba214448b087cf163e74c75a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37425 } } peers { permanent_uuid: "c60634e3d40f4fdb9b890b2d27481143" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38433 } }
I20260924 05:01:29.538959 15685 raft_consensus.cc:729] T c3828df18d3a4774a0053986b0f1baae P 42765acdc69b496cba88f5821ca315cc [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42765acdc69b496cba88f5821ca315cc, State: Initialized, Role: FOLLOWER
I20260924 05:01:29.539074 15685 consensus_queue.cc:260] T c3828df18d3a4774a0053986b0f1baae P 42765acdc69b496cba88f5821ca315cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42765acdc69b496cba88f5821ca315cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43015 } } peers { permanent_uuid: "fdfbafe3ba214448b087cf163e74c75a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37425 } } peers { permanent_uuid: "c60634e3d40f4fdb9b890b2d27481143" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38433 } }
I20260924 05:01:29.539299 15685 ts_tablet_manager.cc:1428] T c3828df18d3a4774a0053986b0f1baae P 42765acdc69b496cba88f5821ca315cc: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260924 05:01:29.539541 15368 tablet_copy_service.cc:342] P c60634e3d40f4fdb9b890b2d27481143: Request end of tablet copy session 42765acdc69b496cba88f5821ca315cc-c3828df18d3a4774a0053986b0f1baae received from {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:44437
I20260924 05:01:29.539557 15687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:29.539471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43939 (local address 127.0.98.3:55809)
0924 05:01:29.539501 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:29.539530 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
I20260924 05:01:29.539592 15368 tablet_copy_service.cc:433] P c60634e3d40f4fdb9b890b2d27481143: ending tablet copy session 42765acdc69b496cba88f5821ca315cc-c3828df18d3a4774a0053986b0f1baae on tablet c3828df18d3a4774a0053986b0f1baae with peer 42765acdc69b496cba88f5821ca315cc
I20260924 05:01:29.539924 15687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:29.539833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43939 (local address 127.0.98.3:38027)
0924 05:01:29.539878 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:29.539901 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260924 05:01:29.904790 15691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:29.904591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43939 (local address 127.0.98.2:38323)
0924 05:01:29.904699 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:29.904731 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":57,"threads_started":1}
I20260924 05:01:29.904918 15692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:29.904721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43939 (local address 127.0.98.1:60303)
0924 05:01:29.904829 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:29.904861 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":43,"threads_started":1}
I20260924 05:01:30.036751 15692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:30.036622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43939 (local address 127.0.98.1:43363)
0924 05:01:30.036677 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:30.036711 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260924 05:01:30.036793 15617 raft_consensus.cc:1203] T c3828df18d3a4774a0053986b0f1baae P 42765acdc69b496cba88f5821ca315cc [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260924 05:01:30.037251 15687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:30.037120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43939 (local address 127.0.98.3:43549)
0924 05:01:30.037167 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:30.037201 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260924 05:01:30.905489 15695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:30.905226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43939 (local address 127.0.98.2:58199)
0924 05:01:30.905362 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:30.905398 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":59,"threads_started":1}
I20260924 05:01:31.037407 15696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:31.037188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43939 (local address 127.0.98.1:48575)
0924 05:01:31.037315 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:31.037357 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":61,"threads_started":1}
I20260924 05:01:31.037889 15697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:31.037666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43939 (local address 127.0.98.3:42531)
0924 05:01:31.037797 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:31.037835 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":75,"threads_started":1}
I20260924 05:01:31.906196 15698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:31.905979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43939 (local address 127.0.98.2:32947)
0924 05:01:31.906106 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:31.906148 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":67,"threads_started":1}
I20260924 05:01:32.038213 15699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:32.037961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43939 (local address 127.0.98.1:32789)
0924 05:01:32.038122 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:32.038166 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":60,"threads_started":1}
I20260924 05:01:32.038538 15700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:32.038289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43939 (local address 127.0.98.3:43443)
0924 05:01:32.038462 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:32.038494 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":64,"threads_started":1}
I20260924 05:01:32.038612 15699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:32.038502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43939 (local address 127.0.98.1:35395)
0924 05:01:32.038551 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:32.038576 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260924 05:01:32.906927 15701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:32.906680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43939 (local address 127.0.98.2:54125)
0924 05:01:32.906820 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:32.906879 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":94,"threads_started":1}
I20260924 05:01:33.039299 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:33.039044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43939 (local address 127.0.98.3:47415)
0924 05:01:33.039185 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:33.039247 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":58,"threads_started":1}
I20260924 05:01:33.039366 15703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:33.039084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43939 (local address 127.0.98.1:49441)
0924 05:01:33.039273 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:33.039313 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":115,"threads_started":1}
I20260924 05:01:33.907562 15704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:33.907337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43939 (local address 127.0.98.2:38047)
0924 05:01:33.907478 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:33.907509 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":82,"threads_started":1}
I20260924 05:01:34.040051 15705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:34.039825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43939 (local address 127.0.98.3:37289)
0924 05:01:34.039958 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:34.039994 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":65,"threads_started":1}
I20260924 05:01:34.040050 15706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:34.039820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43939 (local address 127.0.98.1:57325)
0924 05:01:34.039964 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:34.040001 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":71,"threads_started":1}
I20260924 05:01:34.908267 15707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:34.908005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43939 (local address 127.0.98.2:39171)
0924 05:01:34.908145 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:34.908189 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":63,"threads_started":1}
I20260924 05:01:35.040663 15709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:35.040471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43939 (local address 127.0.98.1:52959)
0924 05:01:35.040587 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:35.040617 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":50,"threads_started":1}
I20260924 05:01:35.040664 15708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:35.040454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43939 (local address 127.0.98.3:41069)
0924 05:01:35.040577 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:35.040617 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":61,"threads_started":1}
I20260924 05:01:35.908913 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:35.908710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43939 (local address 127.0.98.2:51165)
0924 05:01:35.908828 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:35.908864 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":64,"threads_started":1}
I20260924 05:01:36.041319 15712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:36.041113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43939 (local address 127.0.98.3:35145)
0924 05:01:36.041240 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:36.041271 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":63,"threads_started":1}
I20260924 05:01:36.041319 15711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:36.041105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43939 (local address 127.0.98.1:43285)
0924 05:01:36.041239 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:36.041271 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":66,"threads_started":1}
I20260924 05:01:36.909624 15713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:36.909342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43939 (local address 127.0.98.2:54329)
0924 05:01:36.909538 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:36.909577 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":114,"threads_started":1}
I20260924 05:01:37.042100 15715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:37.041877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43939 (local address 127.0.98.1:49381)
0924 05:01:37.042012 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:37.042046 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":63,"threads_started":1}
I20260924 05:01:37.042100 15714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:37.041856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43939 (local address 127.0.98.3:58635)
0924 05:01:37.042007 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:37.042045 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":58,"threads_started":1}
I20260924 05:01:37.910240 15716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:37.910040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43939 (local address 127.0.98.2:49049)
0924 05:01:37.910153 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:37.910190 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":59,"threads_started":1}
I20260924 05:01:38.042877 15717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:38.042635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43939 (local address 127.0.98.3:43811)
0924 05:01:38.042786 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:38.042826 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":101,"threads_started":1}
I20260924 05:01:38.042925 15718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:38.042612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43939 (local address 127.0.98.1:39873)
0924 05:01:38.042834 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:38.042879 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":84,"threads_started":1}
I20260924 05:01:38.910857 15719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:38.910633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43939 (local address 127.0.98.2:58455)
0924 05:01:38.910758 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:38.910793 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":63,"threads_started":1}
I20260924 05:01:39.043572 15721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:39.043349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43939 (local address 127.0.98.1:41045)
0924 05:01:39.043492 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:39.043526 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":66,"threads_started":1}
I20260924 05:01:39.043574 15720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:39.043350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43939 (local address 127.0.98.3:48209)
0924 05:01:39.043490 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:39.043528 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":60,"threads_started":1}
I20260924 05:01:39.911497 15722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:39.911284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43939 (local address 127.0.98.2:60313)
0924 05:01:39.911404 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:39.911448 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":58,"threads_started":1}
I20260924 05:01:40.044242 15723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:40.044037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43939 (local address 127.0.98.1:53717)
0924 05:01:40.044158 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:40.044192 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":58,"threads_started":1}
I20260924 05:01:40.044242 15724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:40.044037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43939 (local address 127.0.98.3:49531)
0924 05:01:40.044162 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:40.044195 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":55,"threads_started":1}
I20260924 05:01:40.912075 15725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:40.911886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43939 (local address 127.0.98.2:53561)
0924 05:01:40.911994 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:40.912027 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":59,"threads_started":1}
I20260924 05:01:41.044899 15726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:41.044681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43939 (local address 127.0.98.1:57743)
0924 05:01:41.044816 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:41.044849 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":79,"threads_started":1}
I20260924 05:01:41.044987 15727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:41.044758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43939 (local address 127.0.98.3:52843)
0924 05:01:41.044889 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:41.044924 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":53,"threads_started":1}
I20260924 05:01:41.912796 15728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:41.912530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43939 (local address 127.0.98.2:49957)
0924 05:01:41.912686 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:41.912744 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":74,"threads_started":1}
I20260924 05:01:41.987581 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 15235
I20260924 05:01:41.991904 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 15399
I20260924 05:01:41.995558 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 15532
2026-09-24T05:01:41Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18535 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:01:42 dist-test-slave-70st krb5kdc[15731](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:01:42 dist-test-slave-70st krb5kdc[15731](info): set up 2 sockets
Sep 24 05:01:42 dist-test-slave-70st krb5kdc[15731](info): commencing operation
krb5kdc: starting...
W20260924 05:01:42.941216 392 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/krb5kdc/test-user.keytab.
Sep 24 05:01:43 dist-test-slave-70st krb5kdc[15731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226103, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:01:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:01:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 05:01:43.054432 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41625
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41625
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:01:43.132560 15744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:01:43.132767 15744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:01:43.132797 15744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:01:43.133967 15744 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 05:01:43.134006 15744 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 05:01:43.134022 15744 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 05:01:43.134037 15744 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:01:43.134052 15744 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:01:43.134070 15744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:01:43.134093 15744 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:01:43.134116 15744 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:01:43.135599 15744 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41625
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15744
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:01:43.135842 15744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:01:43.136045 15744 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:01:43.140818 15750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:01:43.140833 15749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:01:43.141016 15744 server_base.cc:1034] running on GCE node
W20260924 05:01:43.140836 15752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:01:43.141321 15744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:01:43.141602 15744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:01:43.142746 15744 hybrid_clock.cc:648] HybridClock initialized: now 1790226103142728 us; error 34 us; skew 500 ppm
Sep 24 05:01:43 dist-test-slave-70st krb5kdc[15731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226103, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:01:43.145004 15744 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 05:01:43.145481 15744 webserver.cc:460] Webserver started at http://127.0.98.62:43227/ using document root <none> and password file <none>
I20260924 05:01:43.145687 15744 fs_manager.cc:359] Metadata directory not provided
I20260924 05:01:43.145730 15744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:01:43.145848 15744 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:01:43.146848 15744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/master-0/data/instance:
uuid: "e45cc94f6d2f434f9e8cf8dd78e8dff7"
format_stamp: "Formatted at 2026-09-24 05:01:43 on dist-test-slave-70st"
server_key: "bc285d33076a5410016734cc94a222b3"
server_key_iv: "9a02138c971c7442b414f0d78a1bcd86"
server_key_version: "encryptionkey@0"
I20260924 05:01:43.147167 15744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/master-0/wal/instance:
uuid: "e45cc94f6d2f434f9e8cf8dd78e8dff7"
format_stamp: "Formatted at 2026-09-24 05:01:43 on dist-test-slave-70st"
server_key: "bc285d33076a5410016734cc94a222b3"
server_key_iv: "9a02138c971c7442b414f0d78a1bcd86"
server_key_version: "encryptionkey@0"
I20260924 05:01:43.148373 15744 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 05:01:43.149008 15759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:01:43.149138 15744 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 05:01:43.149210 15744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/master-0/wal
uuid: "e45cc94f6d2f434f9e8cf8dd78e8dff7"
format_stamp: "Formatted at 2026-09-24 05:01:43 on dist-test-slave-70st"
server_key: "bc285d33076a5410016734cc94a222b3"
server_key_iv: "9a02138c971c7442b414f0d78a1bcd86"
server_key_version: "encryptionkey@0"
I20260924 05:01:43.149282 15744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:01:43.169142 15744 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:01:43.171025 15744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:01:43.171139 15744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:01:43.174932 15744 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41625
I20260924 05:01:43.174990 15811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41625 every 8 connection(s)
I20260924 05:01:43.175323 15744 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:01:43.176129 15812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:01:43.178395 15812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e45cc94f6d2f434f9e8cf8dd78e8dff7: Bootstrap starting.
I20260924 05:01:43.178808 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 15744
I20260924 05:01:43.178890 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/master-0/wal/instance
I20260924 05:01:43.179016 15812 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e45cc94f6d2f434f9e8cf8dd78e8dff7: Neither blocks nor log segments found. Creating new log.
I20260924 05:01:43.179082 392 external_mini_cluster.cc:1428] Setting key 960277192d407e3a2b4d1ee6be880899
I20260924 05:01:43.179307 15812 log.cc:824] T 00000000000000000000000000000000 P e45cc94f6d2f434f9e8cf8dd78e8dff7: Log is configured to *not* fsync() on all Append() calls
I20260924 05:01:43.180056 15812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e45cc94f6d2f434f9e8cf8dd78e8dff7: No bootstrap required, opened a new log
I20260924 05:01:43.182314 15812 raft_consensus.cc:348] T 00000000000000000000000000000000 P e45cc94f6d2f434f9e8cf8dd78e8dff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e45cc94f6d2f434f9e8cf8dd78e8dff7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41625 } }
I20260924 05:01:43.182475 15812 raft_consensus.cc:374] T 00000000000000000000000000000000 P e45cc94f6d2f434f9e8cf8dd78e8dff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:01:43.182503 15812 raft_consensus.cc:729] T 00000000000000000000000000000000 P e45cc94f6d2f434f9e8cf8dd78e8dff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e45cc94f6d2f434f9e8cf8dd78e8dff7, State: Initialized, Role: FOLLOWER
I20260924 05:01:43.182627 15812 consensus_queue.cc:260] T 00000000000000000000000000000000 P e45cc94f6d2f434f9e8cf8dd78e8dff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e45cc94f6d2f434f9e8cf8dd78e8dff7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41625 } }
I20260924 05:01:43.182708 15812 raft_consensus.cc:388] T 00000000000000000000000000000000 P e45cc94f6d2f434f9e8cf8dd78e8dff7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:01:43.182741 15812 raft_consensus.cc:482] T 00000000000000000000000000000000 P e45cc94f6d2f434f9e8cf8dd78e8dff7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:01:43.182787 15812 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e45cc94f6d2f434f9e8cf8dd78e8dff7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:01:43.183444 15812 raft_consensus.cc:504] T 00000000000000000000000000000000 P e45cc94f6d2f434f9e8cf8dd78e8dff7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e45cc94f6d2f434f9e8cf8dd78e8dff7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41625 } }
I20260924 05:01:43.183552 15812 leader_election.cc:302] T 00000000000000000000000000000000 P e45cc94f6d2f434f9e8cf8dd78e8dff7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e45cc94f6d2f434f9e8cf8dd78e8dff7; no voters:
I20260924 05:01:43.183712 15812 leader_election.cc:288] T 00000000000000000000000000000000 P e45cc94f6d2f434f9e8cf8dd78e8dff7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:01:43.183753 15817 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e45cc94f6d2f434f9e8cf8dd78e8dff7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:01:43.183981 15817 raft_consensus.cc:686] T 00000000000000000000000000000000 P e45cc94f6d2f434f9e8cf8dd78e8dff7 [term 1 LEADER]: Becoming Leader. State: Replica: e45cc94f6d2f434f9e8cf8dd78e8dff7, State: Running, Role: LEADER
I20260924 05:01:43.183990 15812 sys_catalog.cc:564] T 00000000000000000000000000000000 P e45cc94f6d2f434f9e8cf8dd78e8dff7 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:01:43.184142 15817 consensus_queue.cc:237] T 00000000000000000000000000000000 P e45cc94f6d2f434f9e8cf8dd78e8dff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e45cc94f6d2f434f9e8cf8dd78e8dff7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41625 } }
I20260924 05:01:43.184620 15819 sys_catalog.cc:455] T 00000000000000000000000000000000 P e45cc94f6d2f434f9e8cf8dd78e8dff7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e45cc94f6d2f434f9e8cf8dd78e8dff7. Latest consensus state: current_term: 1 leader_uuid: "e45cc94f6d2f434f9e8cf8dd78e8dff7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e45cc94f6d2f434f9e8cf8dd78e8dff7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41625 } } }
I20260924 05:01:43.184702 15819 sys_catalog.cc:458] T 00000000000000000000000000000000 P e45cc94f6d2f434f9e8cf8dd78e8dff7 [sys.catalog]: This master's current role is: LEADER
I20260924 05:01:43.185614 15818 sys_catalog.cc:455] T 00000000000000000000000000000000 P e45cc94f6d2f434f9e8cf8dd78e8dff7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e45cc94f6d2f434f9e8cf8dd78e8dff7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e45cc94f6d2f434f9e8cf8dd78e8dff7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41625 } } }
I20260924 05:01:43.185688 15818 sys_catalog.cc:458] T 00000000000000000000000000000000 P e45cc94f6d2f434f9e8cf8dd78e8dff7 [sys.catalog]: This master's current role is: LEADER
I20260924 05:01:43.185783 15823 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 24 05:01:43 dist-test-slave-70st krb5kdc[15731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226103, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:01:43.186316 15823 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:01:43.188035 15823 catalog_manager.cc:1347] Generated new cluster ID: 31fbd21556764b1e891939dae73550c7
I20260924 05:01:43.188087 15823 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:01:43.190061 15816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:43.179574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46338 (local address 127.0.98.62:41625)
0924 05:01:43.179736 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 05:01:43.179744 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 05:01:43.179778 (+ 34us) server_negotiation.cc:407] Connection header received
0924 05:01:43.180319 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:01:43.180327 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:01:43.180391 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:01:43.180570 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:01:43.181138 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:43.182558 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:43.183581 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:43.184178 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:43.184615 (+ 437us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:43.186865 (+ 2250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:01:43.186869 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:01:43.186869 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:01:43.186880 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:01:43.188670 (+ 1790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:01:43.188981 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:01:43.188984 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:01:43.188984 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:01:43.189070 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:01:43.189238 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:01:43.189239 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:01:43.189240 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:01:43.189510 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:01:43.189642 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0924 05:01:43.189755 (+ 113us) server_negotiation.cc:299] Negotiation successful
0924 05:01:43.189888 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":73,"threads_started":1}
I20260924 05:01:43.196269 15823 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:01:43.196766 15823 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:01:43.202642 15823 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e45cc94f6d2f434f9e8cf8dd78e8dff7: Generated new TSK 0
I20260924 05:01:43.202800 15823 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 05:01:43.229961 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41625
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:01:43.308698 15838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:01:43.308899 15838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:01:43.308921 15838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:01:43.310060 15838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:01:43.310097 15838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:01:43.310123 15838 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 05:01:43.311511 15838 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15838
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:01:43.311735 15838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:01:43.311940 15838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:01:43.316625 15843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:01:43.316700 15846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:01:43.316774 15844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:01:43.316782 15838 server_base.cc:1034] running on GCE node
I20260924 05:01:43.317031 15838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:01:43.317236 15838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:01:43.318367 15838 hybrid_clock.cc:648] HybridClock initialized: now 1790226103318350 us; error 31 us; skew 500 ppm
Sep 24 05:01:43 dist-test-slave-70st krb5kdc[15731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226103, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:01:43.320601 15838 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 05:01:43.320991 15838 webserver.cc:460] Webserver started at http://127.0.98.1:44521/ using document root <none> and password file <none>
I20260924 05:01:43.321185 15838 fs_manager.cc:359] Metadata directory not provided
I20260924 05:01:43.321223 15838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:01:43.321309 15838 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:01:43.322299 15838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-0/data/instance:
uuid: "6590438bffac4f5d8023294a6d2fd4ff"
format_stamp: "Formatted at 2026-09-24 05:01:43 on dist-test-slave-70st"
server_key: "b4f01fdb4ff7c2a227ab16508d224509"
server_key_iv: "359943cd0bcef6f5b6d2416d8cd8b768"
server_key_version: "encryptionkey@0"
I20260924 05:01:43.322578 15838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-0/wal/instance:
uuid: "6590438bffac4f5d8023294a6d2fd4ff"
format_stamp: "Formatted at 2026-09-24 05:01:43 on dist-test-slave-70st"
server_key: "b4f01fdb4ff7c2a227ab16508d224509"
server_key_iv: "359943cd0bcef6f5b6d2416d8cd8b768"
server_key_version: "encryptionkey@0"
I20260924 05:01:43.323769 15838 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 05:01:43.324417 15853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:01:43.324568 15838 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:01:43.324645 15838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-0/wal
uuid: "6590438bffac4f5d8023294a6d2fd4ff"
format_stamp: "Formatted at 2026-09-24 05:01:43 on dist-test-slave-70st"
server_key: "b4f01fdb4ff7c2a227ab16508d224509"
server_key_iv: "359943cd0bcef6f5b6d2416d8cd8b768"
server_key_version: "encryptionkey@0"
I20260924 05:01:43.324702 15838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:01:43.348014 15838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:01:43.349857 15838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:01:43.350000 15838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:01:43.350216 15838 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:01:43.350518 15838 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:01:43.350554 15838 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:01:43.350584 15838 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:01:43.350620 15838 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:01:43.356035 15838 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33823
I20260924 05:01:43.356099 15966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33823 every 8 connection(s)
I20260924 05:01:43.356376 15838 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 05:01:43 dist-test-slave-70st krb5kdc[15731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226103, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:01:43.364504 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 15838
I20260924 05:01:43.364612 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-0/wal/instance
I20260924 05:01:43.364682 15816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:43.357027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38545 (local address 127.0.98.62:41625)
0924 05:01:43.357085 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 05:01:43.357087 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:01:43.357273 (+ 186us) server_negotiation.cc:407] Connection header received
0924 05:01:43.357788 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:01:43.357789 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:01:43.357799 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:01:43.357851 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:01:43.358747 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:43.359237 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:43.360161 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:43.360343 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:43.360394 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:43.362297 (+ 1903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:01:43.362299 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:01:43.362299 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:01:43.362306 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:01:43.363501 (+ 1195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:01:43.363849 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:01:43.363850 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:01:43.363850 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:01:43.363900 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:01:43.364106 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:01:43.364107 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:01:43.364107 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:01:43.364211 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:01:43.364264 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 05:01:43.364511 (+ 247us) server_negotiation.cc:299] Negotiation successful
0924 05:01:43.364593 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 05:01:43.364820 392 external_mini_cluster.cc:1428] Setting key 9eda35f165dde8880d813c7aa7086f23
I20260924 05:01:43.364841 15969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:43.357007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41625 (local address 127.0.98.1:38545)
0924 05:01:43.357158 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:43.357192 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 05:01:43.357708 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:01:43.357919 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:01:43.357922 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:01:43.358489 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:01:43.358681 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:01:43.358686 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:01:43.359298 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:01:43.359299 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:01:43.360100 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:01:43.360102 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:01:43.360398 (+ 296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:01:43.360398 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:01:43.360505 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:43.360951 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:01:43.360959 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:01:43.362211 (+ 1252us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:01:43.363572 (+ 1361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:01:43.363579 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:01:43.363581 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:01:43.363787 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:01:43.363953 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:01:43.363954 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:01:43.363954 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:01:43.364054 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:01:43.364263 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:01:43.364269 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:01:43.364400 (+ 131us) client_negotiation.cc:769] Sending connection context
0924 05:01:43.364486 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 05:01:43.364635 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":93,"threads_started":1}
I20260924 05:01:43.365303 15967 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41625
I20260924 05:01:43.365401 15967 heartbeater.cc:464] Registering TS with master...
I20260924 05:01:43.365598 15967 heartbeater.cc:511] Master 127.0.98.62:41625 requested a full tablet report, sending...
I20260924 05:01:43.366112 15775 ts_manager.cc:194] Registered new tserver with Master: 6590438bffac4f5d8023294a6d2fd4ff (127.0.98.1:33823)
I20260924 05:01:43.366937 15775 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:38545
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 05:01:43.387924 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41625
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:01:43.467065 15972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:01:43.467253 15972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:01:43.467274 15972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:01:43.468348 15972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:01:43.468387 15972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:01:43.468412 15972 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 05:01:43.469810 15972 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15972
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:01:43.469985 15972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:01:43.470155 15972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:01:43.474891 15978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:01:43.474879 15977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:01:43.474907 15972 server_base.cc:1034] running on GCE node
W20260924 05:01:43.474874 15980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:01:43.475277 15972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:01:43.475522 15972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:01:43.476691 15972 hybrid_clock.cc:648] HybridClock initialized: now 1790226103476676 us; error 25 us; skew 500 ppm
Sep 24 05:01:43 dist-test-slave-70st krb5kdc[15731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226103, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:01:43.479195 15972 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 05:01:43.479624 15972 webserver.cc:460] Webserver started at http://127.0.98.2:44059/ using document root <none> and password file <none>
I20260924 05:01:43.479836 15972 fs_manager.cc:359] Metadata directory not provided
I20260924 05:01:43.479884 15972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:01:43.480010 15972 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:01:43.481033 15972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-1/data/instance:
uuid: "1d219c86d9ea4b6aa6d894eb20867698"
format_stamp: "Formatted at 2026-09-24 05:01:43 on dist-test-slave-70st"
server_key: "9182aeba844e056977f6da6dfdfdbd18"
server_key_iv: "0cc71887ec5726087abc5e4c97c68e26"
server_key_version: "encryptionkey@0"
I20260924 05:01:43.481375 15972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-1/wal/instance:
uuid: "1d219c86d9ea4b6aa6d894eb20867698"
format_stamp: "Formatted at 2026-09-24 05:01:43 on dist-test-slave-70st"
server_key: "9182aeba844e056977f6da6dfdfdbd18"
server_key_iv: "0cc71887ec5726087abc5e4c97c68e26"
server_key_version: "encryptionkey@0"
I20260924 05:01:43.482613 15972 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 05:01:43.483254 15987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:01:43.483428 15972 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:01:43.483500 15972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-1/wal
uuid: "1d219c86d9ea4b6aa6d894eb20867698"
format_stamp: "Formatted at 2026-09-24 05:01:43 on dist-test-slave-70st"
server_key: "9182aeba844e056977f6da6dfdfdbd18"
server_key_iv: "0cc71887ec5726087abc5e4c97c68e26"
server_key_version: "encryptionkey@0"
I20260924 05:01:43.483569 15972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:01:43.506773 15972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:01:43.508553 15972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:01:43.508674 15972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:01:43.508900 15972 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:01:43.509197 15972 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:01:43.509229 15972 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:01:43.509251 15972 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:01:43.509274 15972 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:01:43.514683 15972 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42831
I20260924 05:01:43.514753 16100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42831 every 8 connection(s)
I20260924 05:01:43.515053 15972 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-1/data/info.pb
Sep 24 05:01:43 dist-test-slave-70st krb5kdc[15731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226103, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:01:43.522648 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 15972
I20260924 05:01:43.522769 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-1/wal/instance
I20260924 05:01:43.522974 392 external_mini_cluster.cc:1428] Setting key bba88490ae642f435ddcf047d7d79732
I20260924 05:01:43.523691 15816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:43.515709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54271 (local address 127.0.98.62:41625)
0924 05:01:43.515753 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 05:01:43.515754 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:01:43.515956 (+ 202us) server_negotiation.cc:407] Connection header received
0924 05:01:43.516459 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:01:43.516460 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:01:43.516472 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:01:43.516524 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:01:43.517460 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:43.517964 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:43.518963 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:43.519155 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:43.519209 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:43.521155 (+ 1946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:01:43.521156 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:01:43.521157 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:01:43.521164 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:01:43.522300 (+ 1136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:01:43.522678 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:01:43.522679 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:01:43.522679 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:01:43.522737 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:01:43.523059 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:01:43.523060 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:01:43.523060 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:01:43.523181 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:01:43.523240 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 05:01:43.523519 (+ 279us) server_negotiation.cc:299] Negotiation successful
0924 05:01:43.523602 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 05:01:43.523871 16103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:43.515667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41625 (local address 127.0.98.2:54271)
0924 05:01:43.515845 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:43.515880 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 05:01:43.516366 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:01:43.516595 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:01:43.516599 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:01:43.517156 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:01:43.517347 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:01:43.517352 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:01:43.518055 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:01:43.518056 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:01:43.518884 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:01:43.518886 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:01:43.519226 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:01:43.519227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:01:43.519344 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:43.519798 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:01:43.519807 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:01:43.521069 (+ 1262us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:01:43.522378 (+ 1309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:01:43.522385 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:01:43.522387 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:01:43.522611 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:01:43.522811 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:01:43.522812 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:01:43.522812 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:01:43.522965 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:01:43.523241 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:01:43.523248 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:01:43.523382 (+ 134us) client_negotiation.cc:769] Sending connection context
0924 05:01:43.523480 (+ 98us) client_negotiation.cc:240] Negotiation successful
0924 05:01:43.523636 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":81,"threads_started":1}
I20260924 05:01:43.524296 16101 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41625
I20260924 05:01:43.524377 16101 heartbeater.cc:464] Registering TS with master...
I20260924 05:01:43.524542 16101 heartbeater.cc:511] Master 127.0.98.62:41625 requested a full tablet report, sending...
I20260924 05:01:43.524895 15775 ts_manager.cc:194] Registered new tserver with Master: 1d219c86d9ea4b6aa6d894eb20867698 (127.0.98.2:42831)
I20260924 05:01:43.525264 15775 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:54271
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 05:01:43.544912 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41625
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:01:43.623332 16106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:01:43.623551 16106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:01:43.623574 16106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:01:43.624691 16106 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:01:43.624725 16106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:01:43.624749 16106 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 05:01:43.626204 16106 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16106
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:01:43.626406 16106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:01:43.626595 16106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:01:43.631336 16112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:01:43.631343 16114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:01:43.631343 16111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:01:43.631587 16106 server_base.cc:1034] running on GCE node
I20260924 05:01:43.631776 16106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:01:43.632002 16106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:01:43.633155 16106 hybrid_clock.cc:648] HybridClock initialized: now 1790226103633138 us; error 25 us; skew 500 ppm
Sep 24 05:01:43 dist-test-slave-70st krb5kdc[15731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226103, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:01:43.635581 16106 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 05:01:43.636004 16106 webserver.cc:460] Webserver started at http://127.0.98.3:40245/ using document root <none> and password file <none>
I20260924 05:01:43.636195 16106 fs_manager.cc:359] Metadata directory not provided
I20260924 05:01:43.636242 16106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:01:43.636363 16106 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:01:43.637359 16106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-2/data/instance:
uuid: "e0cc0328540d4a9db537a475b14a4ab3"
format_stamp: "Formatted at 2026-09-24 05:01:43 on dist-test-slave-70st"
server_key: "8e0ce449eaf3849a57093216055d4807"
server_key_iv: "c4d89a7ff885b5cdf2cb5321864d74dc"
server_key_version: "encryptionkey@0"
I20260924 05:01:43.637724 16106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e0cc0328540d4a9db537a475b14a4ab3"
format_stamp: "Formatted at 2026-09-24 05:01:43 on dist-test-slave-70st"
server_key: "8e0ce449eaf3849a57093216055d4807"
server_key_iv: "c4d89a7ff885b5cdf2cb5321864d74dc"
server_key_version: "encryptionkey@0"
I20260924 05:01:43.638873 16106 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 05:01:43.639513 16121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:01:43.639657 16106 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 05:01:43.639734 16106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-2/wal
uuid: "e0cc0328540d4a9db537a475b14a4ab3"
format_stamp: "Formatted at 2026-09-24 05:01:43 on dist-test-slave-70st"
server_key: "8e0ce449eaf3849a57093216055d4807"
server_key_iv: "c4d89a7ff885b5cdf2cb5321864d74dc"
server_key_version: "encryptionkey@0"
I20260924 05:01:43.639798 16106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:01:43.653010 16106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:01:43.654747 16106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:01:43.654873 16106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:01:43.655102 16106 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:01:43.655378 16106 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:01:43.655411 16106 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:01:43.655439 16106 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:01:43.655464 16106 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:01:43.660656 16106 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45917
I20260924 05:01:43.660714 16234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45917 every 8 connection(s)
I20260924 05:01:43.660995 16106 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 05:01:43 dist-test-slave-70st krb5kdc[15731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226103, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:01:43.669057 15816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:43.661706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44665 (local address 127.0.98.62:41625)
0924 05:01:43.661765 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 05:01:43.661767 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:01:43.661915 (+ 148us) server_negotiation.cc:407] Connection header received
0924 05:01:43.662376 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:01:43.662377 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:01:43.662386 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:01:43.662424 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:01:43.663286 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:43.663771 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:43.664721 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:43.664894 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:43.664948 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:43.666972 (+ 2024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:01:43.666973 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:01:43.666973 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:01:43.666979 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:01:43.668052 (+ 1073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:01:43.668380 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:01:43.668380 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:01:43.668381 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:01:43.668420 (+ 39us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:01:43.668597 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:01:43.668597 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:01:43.668597 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:01:43.668687 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:01:43.668728 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0924 05:01:43.668937 (+ 209us) server_negotiation.cc:299] Negotiation successful
0924 05:01:43.668991 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 05:01:43.669229 16237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:43.661637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41625 (local address 127.0.98.3:44665)
0924 05:01:43.661804 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:43.661840 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 05:01:43.662294 (+ 454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:01:43.662480 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:01:43.662483 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:01:43.663036 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:01:43.663222 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:01:43.663227 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:01:43.663844 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:01:43.663844 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:01:43.664659 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:01:43.664662 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:01:43.664973 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:01:43.664974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:01:43.665084 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:43.665603 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:01:43.665611 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:01:43.666893 (+ 1282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:01:43.668116 (+ 1223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:01:43.668121 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:01:43.668122 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:01:43.668325 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:01:43.668469 (+ 144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:01:43.668469 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:01:43.668469 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:01:43.668557 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:01:43.668726 (+ 169us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:01:43.668728 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:01:43.668838 (+ 110us) client_negotiation.cc:769] Sending connection context
0924 05:01:43.668913 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 05:01:43.669036 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":71,"threads_started":1}
I20260924 05:01:43.669332 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 16106
I20260924 05:01:43.669478 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-2/wal/instance
I20260924 05:01:43.669667 392 external_mini_cluster.cc:1428] Setting key a426ce63c0d9aeb07d23183c2f77622d
I20260924 05:01:43.669708 16235 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41625
I20260924 05:01:43.669799 16235 heartbeater.cc:464] Registering TS with master...
I20260924 05:01:43.669983 16235 heartbeater.cc:511] Master 127.0.98.62:41625 requested a full tablet report, sending...
I20260924 05:01:43.670346 15776 ts_manager.cc:194] Registered new tserver with Master: e0cc0328540d4a9db537a475b14a4ab3 (127.0.98.3:45917)
I20260924 05:01:43.670727 15776 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:44665
I20260924 05:01:43.681206 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 05:01:43.688198 15816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:43.682570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46354 (local address 127.0.98.62:41625)
0924 05:01:43.682624 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 05:01:43.682626 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:01:43.682711 (+ 85us) server_negotiation.cc:407] Connection header received
0924 05:01:43.682767 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:01:43.682768 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:01:43.682777 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:01:43.682820 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:01:43.683343 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:43.683905 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:43.684632 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:43.684848 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:43.684900 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:43.686034 (+ 1134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:01:43.686035 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:01:43.686036 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:01:43.686042 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:01:43.687163 (+ 1121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:01:43.687489 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:01:43.687490 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:01:43.687490 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:01:43.687543 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:01:43.687700 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:01:43.687701 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:01:43.687701 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:01:43.687839 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:01:43.687936 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 05:01:43.688032 (+ 96us) server_negotiation.cc:299] Negotiation successful
0924 05:01:43.688111 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 05:01:43.689631 15776 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46354:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 05:01:43.691149 15776 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 05:01:43.697021 16248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:43.693354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38880 (local address 127.0.98.2:42831)
0924 05:01:43.693708 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0924 05:01:43.693712 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 05:01:43.693721 (+ 9us) server_negotiation.cc:407] Connection header received
0924 05:01:43.693758 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:01:43.693760 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:01:43.693774 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:01:43.693993 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:01:43.694049 (+ 56us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:43.694712 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:43.696107 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:43.696644 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:43.696704 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:43.696763 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 05:01:43.696898 (+ 135us) server_negotiation.cc:299] Negotiation successful
0924 05:01:43.696945 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":106,"threads_started":1}
I20260924 05:01:43.697001 16243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:43.693245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42831 (local address 127.0.0.1:38880)
0924 05:01:43.693564 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:43.693572 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 05:01:43.693605 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:01:43.693862 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:01:43.693862 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:01:43.693865 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:01:43.693925 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:01:43.693927 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:01:43.694795 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:01:43.694796 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:01:43.696015 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:01:43.696017 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:01:43.696716 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:01:43.696717 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:01:43.696798 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:43.696813 (+ 15us) client_negotiation.cc:769] Sending connection context
0924 05:01:43.696876 (+ 63us) client_negotiation.cc:240] Negotiation successful
0924 05:01:43.696927 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":31,"threads_started":1}
I20260924 05:01:43.697175 16245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:43.693154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41050 (local address 127.0.98.3:45917)
0924 05:01:43.693458 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0924 05:01:43.693462 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 05:01:43.693613 (+ 151us) server_negotiation.cc:407] Connection header received
0924 05:01:43.693689 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:01:43.693691 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:01:43.693707 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:01:43.693774 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:01:43.693979 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:43.695044 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:43.696073 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:43.696704 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:43.696769 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:43.696822 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 05:01:43.697054 (+ 232us) server_negotiation.cc:299] Negotiation successful
0924 05:01:43.697094 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":85,"threads_started":1}
I20260924 05:01:43.697230 16246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:43.693068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45917 (local address 127.0.0.1:41050)
0924 05:01:43.693543 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:43.693552 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:01:43.693605 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:01:43.693850 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:01:43.693851 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:01:43.693853 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:01:43.693891 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:01:43.693893 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:01:43.695140 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:01:43.695141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:01:43.695999 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:01:43.696001 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:01:43.696856 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:01:43.696857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:01:43.696919 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:43.696923 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 05:01:43.696970 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 05:01:43.697010 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"mutex_wait_us":3,"thread_start_us":46,"threads_started":1}
I20260924 05:01:43.697320 16244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:43.693067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33823 (local address 127.0.0.1:33086)
0924 05:01:43.693332 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:43.693372 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 05:01:43.693470 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:01:43.693733 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:01:43.693736 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:01:43.693739 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:01:43.693802 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:01:43.693804 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:01:43.695124 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:01:43.695125 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:01:43.696221 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:01:43.696222 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:01:43.697097 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:01:43.697098 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:01:43.697163 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:43.697166 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:01:43.697210 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 05:01:43.697245 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"mutex_wait_us":39,"thread_start_us":51,"threads_started":1}
I20260924 05:01:43.697576 16247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:43.693240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33086 (local address 127.0.98.1:33823)
0924 05:01:43.693400 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0924 05:01:43.693406 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 05:01:43.693454 (+ 48us) server_negotiation.cc:407] Connection header received
0924 05:01:43.693547 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:01:43.693549 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:01:43.693562 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:01:43.693636 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:01:43.693877 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:43.695023 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:43.696294 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:43.696963 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:43.697016 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:43.697068 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 05:01:43.697458 (+ 390us) server_negotiation.cc:299] Negotiation successful
0924 05:01:43.697498 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":92,"threads_started":1}
I20260924 05:01:43.698011 16169 tablet_service.cc:1467] Processing CreateTablet for tablet 2f7c2974b484419b8125d5a0d97d311b (DEFAULT_TABLE table=test-table [id=677ab201edee4befae64df9fc951a4ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:01:43.698031 16035 tablet_service.cc:1467] Processing CreateTablet for tablet 2f7c2974b484419b8125d5a0d97d311b (DEFAULT_TABLE table=test-table [id=677ab201edee4befae64df9fc951a4ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:01:43.698361 16035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f7c2974b484419b8125d5a0d97d311b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:01:43.698361 16169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f7c2974b484419b8125d5a0d97d311b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:01:43.698403 15901 tablet_service.cc:1467] Processing CreateTablet for tablet 2f7c2974b484419b8125d5a0d97d311b (DEFAULT_TABLE table=test-table [id=677ab201edee4befae64df9fc951a4ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:01:43.698714 15901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f7c2974b484419b8125d5a0d97d311b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:01:43.701017 16249 tablet_bootstrap.cc:492] T 2f7c2974b484419b8125d5a0d97d311b P 6590438bffac4f5d8023294a6d2fd4ff: Bootstrap starting.
I20260924 05:01:43.701565 16250 tablet_bootstrap.cc:492] T 2f7c2974b484419b8125d5a0d97d311b P e0cc0328540d4a9db537a475b14a4ab3: Bootstrap starting.
I20260924 05:01:43.701756 16249 tablet_bootstrap.cc:654] T 2f7c2974b484419b8125d5a0d97d311b P 6590438bffac4f5d8023294a6d2fd4ff: Neither blocks nor log segments found. Creating new log.
I20260924 05:01:43.701741 16251 tablet_bootstrap.cc:492] T 2f7c2974b484419b8125d5a0d97d311b P 1d219c86d9ea4b6aa6d894eb20867698: Bootstrap starting.
I20260924 05:01:43.702337 16249 log.cc:824] T 2f7c2974b484419b8125d5a0d97d311b P 6590438bffac4f5d8023294a6d2fd4ff: Log is configured to *not* fsync() on all Append() calls
I20260924 05:01:43.702366 16251 tablet_bootstrap.cc:654] T 2f7c2974b484419b8125d5a0d97d311b P 1d219c86d9ea4b6aa6d894eb20867698: Neither blocks nor log segments found. Creating new log.
I20260924 05:01:43.702853 16251 log.cc:824] T 2f7c2974b484419b8125d5a0d97d311b P 1d219c86d9ea4b6aa6d894eb20867698: Log is configured to *not* fsync() on all Append() calls
I20260924 05:01:43.703089 16249 tablet_bootstrap.cc:492] T 2f7c2974b484419b8125d5a0d97d311b P 6590438bffac4f5d8023294a6d2fd4ff: No bootstrap required, opened a new log
I20260924 05:01:43.703159 16249 ts_tablet_manager.cc:1397] T 2f7c2974b484419b8125d5a0d97d311b P 6590438bffac4f5d8023294a6d2fd4ff: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260924 05:01:43.703490 16251 tablet_bootstrap.cc:492] T 2f7c2974b484419b8125d5a0d97d311b P 1d219c86d9ea4b6aa6d894eb20867698: No bootstrap required, opened a new log
I20260924 05:01:43.703547 16251 ts_tablet_manager.cc:1397] T 2f7c2974b484419b8125d5a0d97d311b P 1d219c86d9ea4b6aa6d894eb20867698: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 05:01:43.703671 16250 tablet_bootstrap.cc:654] T 2f7c2974b484419b8125d5a0d97d311b P e0cc0328540d4a9db537a475b14a4ab3: Neither blocks nor log segments found. Creating new log.
I20260924 05:01:43.704227 16250 log.cc:824] T 2f7c2974b484419b8125d5a0d97d311b P e0cc0328540d4a9db537a475b14a4ab3: Log is configured to *not* fsync() on all Append() calls
I20260924 05:01:43.704947 16250 tablet_bootstrap.cc:492] T 2f7c2974b484419b8125d5a0d97d311b P e0cc0328540d4a9db537a475b14a4ab3: No bootstrap required, opened a new log
I20260924 05:01:43.705019 16250 ts_tablet_manager.cc:1397] T 2f7c2974b484419b8125d5a0d97d311b P e0cc0328540d4a9db537a475b14a4ab3: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260924 05:01:43.705067 16249 raft_consensus.cc:348] T 2f7c2974b484419b8125d5a0d97d311b P 6590438bffac4f5d8023294a6d2fd4ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cc0328540d4a9db537a475b14a4ab3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45917 } } peers { permanent_uuid: "6590438bffac4f5d8023294a6d2fd4ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33823 } } peers { permanent_uuid: "1d219c86d9ea4b6aa6d894eb20867698" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42831 } }
I20260924 05:01:43.705242 16249 raft_consensus.cc:374] T 2f7c2974b484419b8125d5a0d97d311b P 6590438bffac4f5d8023294a6d2fd4ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:01:43.705180 16251 raft_consensus.cc:348] T 2f7c2974b484419b8125d5a0d97d311b P 1d219c86d9ea4b6aa6d894eb20867698 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cc0328540d4a9db537a475b14a4ab3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45917 } } peers { permanent_uuid: "6590438bffac4f5d8023294a6d2fd4ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33823 } } peers { permanent_uuid: "1d219c86d9ea4b6aa6d894eb20867698" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42831 } }
I20260924 05:01:43.705307 16249 raft_consensus.cc:729] T 2f7c2974b484419b8125d5a0d97d311b P 6590438bffac4f5d8023294a6d2fd4ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6590438bffac4f5d8023294a6d2fd4ff, State: Initialized, Role: FOLLOWER
I20260924 05:01:43.705363 16251 raft_consensus.cc:374] T 2f7c2974b484419b8125d5a0d97d311b P 1d219c86d9ea4b6aa6d894eb20867698 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:01:43.705439 16249 consensus_queue.cc:260] T 2f7c2974b484419b8125d5a0d97d311b P 6590438bffac4f5d8023294a6d2fd4ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cc0328540d4a9db537a475b14a4ab3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45917 } } peers { permanent_uuid: "6590438bffac4f5d8023294a6d2fd4ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33823 } } peers { permanent_uuid: "1d219c86d9ea4b6aa6d894eb20867698" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42831 } }
I20260924 05:01:43.705477 16251 raft_consensus.cc:729] T 2f7c2974b484419b8125d5a0d97d311b P 1d219c86d9ea4b6aa6d894eb20867698 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d219c86d9ea4b6aa6d894eb20867698, State: Initialized, Role: FOLLOWER
I20260924 05:01:43.705624 16251 consensus_queue.cc:260] T 2f7c2974b484419b8125d5a0d97d311b P 1d219c86d9ea4b6aa6d894eb20867698 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cc0328540d4a9db537a475b14a4ab3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45917 } } peers { permanent_uuid: "6590438bffac4f5d8023294a6d2fd4ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33823 } } peers { permanent_uuid: "1d219c86d9ea4b6aa6d894eb20867698" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42831 } }
I20260924 05:01:43.705701 16249 ts_tablet_manager.cc:1428] T 2f7c2974b484419b8125d5a0d97d311b P 6590438bffac4f5d8023294a6d2fd4ff: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260924 05:01:43.705834 15967 heartbeater.cc:503] Master 127.0.98.62:41625 was elected leader, sending a full tablet report...
I20260924 05:01:43.705875 16101 heartbeater.cc:503] Master 127.0.98.62:41625 was elected leader, sending a full tablet report...
I20260924 05:01:43.706106 16251 ts_tablet_manager.cc:1428] T 2f7c2974b484419b8125d5a0d97d311b P 1d219c86d9ea4b6aa6d894eb20867698: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260924 05:01:43.706960 16250 raft_consensus.cc:348] T 2f7c2974b484419b8125d5a0d97d311b P e0cc0328540d4a9db537a475b14a4ab3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cc0328540d4a9db537a475b14a4ab3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45917 } } peers { permanent_uuid: "6590438bffac4f5d8023294a6d2fd4ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33823 } } peers { permanent_uuid: "1d219c86d9ea4b6aa6d894eb20867698" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42831 } }
I20260924 05:01:43.707106 16250 raft_consensus.cc:374] T 2f7c2974b484419b8125d5a0d97d311b P e0cc0328540d4a9db537a475b14a4ab3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:01:43.707129 16250 raft_consensus.cc:729] T 2f7c2974b484419b8125d5a0d97d311b P e0cc0328540d4a9db537a475b14a4ab3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0cc0328540d4a9db537a475b14a4ab3, State: Initialized, Role: FOLLOWER
I20260924 05:01:43.707232 16250 consensus_queue.cc:260] T 2f7c2974b484419b8125d5a0d97d311b P e0cc0328540d4a9db537a475b14a4ab3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cc0328540d4a9db537a475b14a4ab3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45917 } } peers { permanent_uuid: "6590438bffac4f5d8023294a6d2fd4ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33823 } } peers { permanent_uuid: "1d219c86d9ea4b6aa6d894eb20867698" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42831 } }
I20260924 05:01:43.707484 16235 heartbeater.cc:503] Master 127.0.98.62:41625 was elected leader, sending a full tablet report...
I20260924 05:01:43.707676 16250 ts_tablet_manager.cc:1428] T 2f7c2974b484419b8125d5a0d97d311b P e0cc0328540d4a9db537a475b14a4ab3: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
W20260924 05:01:43.765590 16102 tablet.cc:2369] T 2f7c2974b484419b8125d5a0d97d311b P 1d219c86d9ea4b6aa6d894eb20867698: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 05:01:43.793681 16257 raft_consensus.cc:482] T 2f7c2974b484419b8125d5a0d97d311b P e0cc0328540d4a9db537a475b14a4ab3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 05:01:43.793820 16257 raft_consensus.cc:504] T 2f7c2974b484419b8125d5a0d97d311b P e0cc0328540d4a9db537a475b14a4ab3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cc0328540d4a9db537a475b14a4ab3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45917 } } peers { permanent_uuid: "6590438bffac4f5d8023294a6d2fd4ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33823 } } peers { permanent_uuid: "1d219c86d9ea4b6aa6d894eb20867698" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42831 } }
I20260924 05:01:43.794166 16257 leader_election.cc:288] T 2f7c2974b484419b8125d5a0d97d311b P e0cc0328540d4a9db537a475b14a4ab3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6590438bffac4f5d8023294a6d2fd4ff (127.0.98.1:33823), 1d219c86d9ea4b6aa6d894eb20867698 (127.0.98.2:42831)
I20260924 05:01:43.797809 16248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:43.794532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48989 (local address 127.0.98.2:42831)
0924 05:01:43.794595 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 05:01:43.794597 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:01:43.794716 (+ 119us) server_negotiation.cc:407] Connection header received
0924 05:01:43.794769 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:01:43.794770 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:01:43.794782 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:01:43.794831 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:01:43.794939 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:43.795616 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:43.796865 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:43.797455 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:43.797537 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:43.797549 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0924 05:01:43.797705 (+ 156us) server_negotiation.cc:299] Negotiation successful
0924 05:01:43.797739 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 05:01:43.797809 16247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:43.794428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46925 (local address 127.0.98.1:33823)
0924 05:01:43.794491 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 05:01:43.794493 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:01:43.794502 (+ 9us) server_negotiation.cc:407] Connection header received
0924 05:01:43.794529 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:01:43.794530 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:01:43.794542 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:01:43.794594 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:01:43.794750 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:43.795521 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:43.796867 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:43.797488 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:43.797542 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:43.797550 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 05:01:43.797701 (+ 151us) server_negotiation.cc:299] Negotiation successful
0924 05:01:43.797739 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 05:01:43.797828 16237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:43.794345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33823 (local address 127.0.98.3:46925)
0924 05:01:43.794396 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:43.794408 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 05:01:43.794460 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:01:43.794608 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:01:43.794609 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:01:43.794612 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:01:43.794684 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:01:43.794686 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:01:43.795594 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:01:43.795595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:01:43.796781 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:01:43.796784 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:01:43.797544 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:01:43.797545 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:01:43.797633 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:43.797635 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 05:01:43.797688 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 05:01:43.797732 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":3}
I20260924 05:01:43.797842 16258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:43.794471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42831 (local address 127.0.98.3:48989)
0924 05:01:43.794649 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:43.794659 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:01:43.794718 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:01:43.794838 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:01:43.794839 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:01:43.794840 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:01:43.794887 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:01:43.794888 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:01:43.795679 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:01:43.795680 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:01:43.796781 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:01:43.796784 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:01:43.797544 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:01:43.797545 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:01:43.797630 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:43.797635 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 05:01:43.797688 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 05:01:43.797732 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"mutex_wait_us":14,"thread_start_us":108,"threads_started":1}
I20260924 05:01:43.798198 15921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f7c2974b484419b8125d5a0d97d311b" candidate_uuid: "e0cc0328540d4a9db537a475b14a4ab3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6590438bffac4f5d8023294a6d2fd4ff" is_pre_election: true
I20260924 05:01:43.798363 15921 raft_consensus.cc:2454] T 2f7c2974b484419b8125d5a0d97d311b P 6590438bffac4f5d8023294a6d2fd4ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0cc0328540d4a9db537a475b14a4ab3 in term 0.
I20260924 05:01:43.798440 16055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f7c2974b484419b8125d5a0d97d311b" candidate_uuid: "e0cc0328540d4a9db537a475b14a4ab3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d219c86d9ea4b6aa6d894eb20867698" is_pre_election: true
I20260924 05:01:43.798594 16055 raft_consensus.cc:2454] T 2f7c2974b484419b8125d5a0d97d311b P 1d219c86d9ea4b6aa6d894eb20867698 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0cc0328540d4a9db537a475b14a4ab3 in term 0.
I20260924 05:01:43.798585 16124 leader_election.cc:302] T 2f7c2974b484419b8125d5a0d97d311b P e0cc0328540d4a9db537a475b14a4ab3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6590438bffac4f5d8023294a6d2fd4ff, e0cc0328540d4a9db537a475b14a4ab3; no voters:
I20260924 05:01:43.798697 16257 raft_consensus.cc:2781] T 2f7c2974b484419b8125d5a0d97d311b P e0cc0328540d4a9db537a475b14a4ab3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 05:01:43.798763 16257 raft_consensus.cc:482] T 2f7c2974b484419b8125d5a0d97d311b P e0cc0328540d4a9db537a475b14a4ab3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 05:01:43.798789 16257 raft_consensus.cc:3037] T 2f7c2974b484419b8125d5a0d97d311b P e0cc0328540d4a9db537a475b14a4ab3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:01:43.799453 16257 raft_consensus.cc:504] T 2f7c2974b484419b8125d5a0d97d311b P e0cc0328540d4a9db537a475b14a4ab3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cc0328540d4a9db537a475b14a4ab3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45917 } } peers { permanent_uuid: "6590438bffac4f5d8023294a6d2fd4ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33823 } } peers { permanent_uuid: "1d219c86d9ea4b6aa6d894eb20867698" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42831 } }
I20260924 05:01:43.799568 16257 leader_election.cc:288] T 2f7c2974b484419b8125d5a0d97d311b P e0cc0328540d4a9db537a475b14a4ab3 [CANDIDATE]: Term 1 election: Requested vote from peers 6590438bffac4f5d8023294a6d2fd4ff (127.0.98.1:33823), 1d219c86d9ea4b6aa6d894eb20867698 (127.0.98.2:42831)
I20260924 05:01:43.799722 16055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f7c2974b484419b8125d5a0d97d311b" candidate_uuid: "e0cc0328540d4a9db537a475b14a4ab3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d219c86d9ea4b6aa6d894eb20867698"
I20260924 05:01:43.799724 15921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f7c2974b484419b8125d5a0d97d311b" candidate_uuid: "e0cc0328540d4a9db537a475b14a4ab3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6590438bffac4f5d8023294a6d2fd4ff"
I20260924 05:01:43.799790 15921 raft_consensus.cc:3037] T 2f7c2974b484419b8125d5a0d97d311b P 6590438bffac4f5d8023294a6d2fd4ff [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:01:43.799791 16055 raft_consensus.cc:3037] T 2f7c2974b484419b8125d5a0d97d311b P 1d219c86d9ea4b6aa6d894eb20867698 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:01:43.800444 16055 raft_consensus.cc:2454] T 2f7c2974b484419b8125d5a0d97d311b P 1d219c86d9ea4b6aa6d894eb20867698 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0cc0328540d4a9db537a475b14a4ab3 in term 1.
I20260924 05:01:43.800491 15921 raft_consensus.cc:2454] T 2f7c2974b484419b8125d5a0d97d311b P 6590438bffac4f5d8023294a6d2fd4ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0cc0328540d4a9db537a475b14a4ab3 in term 1.
I20260924 05:01:43.800628 16124 leader_election.cc:302] T 2f7c2974b484419b8125d5a0d97d311b P e0cc0328540d4a9db537a475b14a4ab3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6590438bffac4f5d8023294a6d2fd4ff, e0cc0328540d4a9db537a475b14a4ab3; no voters:
I20260924 05:01:43.800729 16257 raft_consensus.cc:2781] T 2f7c2974b484419b8125d5a0d97d311b P e0cc0328540d4a9db537a475b14a4ab3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:01:43.800886 16257 raft_consensus.cc:686] T 2f7c2974b484419b8125d5a0d97d311b P e0cc0328540d4a9db537a475b14a4ab3 [term 1 LEADER]: Becoming Leader. State: Replica: e0cc0328540d4a9db537a475b14a4ab3, State: Running, Role: LEADER
I20260924 05:01:43.801046 16257 consensus_queue.cc:237] T 2f7c2974b484419b8125d5a0d97d311b P e0cc0328540d4a9db537a475b14a4ab3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cc0328540d4a9db537a475b14a4ab3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45917 } } peers { permanent_uuid: "6590438bffac4f5d8023294a6d2fd4ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33823 } } peers { permanent_uuid: "1d219c86d9ea4b6aa6d894eb20867698" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42831 } }
I20260924 05:01:43.801831 15775 catalog_manager.cc:5495] T 2f7c2974b484419b8125d5a0d97d311b P e0cc0328540d4a9db537a475b14a4ab3 reported cstate change: term changed from 0 to 1, leader changed from <none> to e0cc0328540d4a9db537a475b14a4ab3 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "e0cc0328540d4a9db537a475b14a4ab3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cc0328540d4a9db537a475b14a4ab3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45917 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6590438bffac4f5d8023294a6d2fd4ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33823 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d219c86d9ea4b6aa6d894eb20867698" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42831 } health_report { overall_health: UNKNOWN } } }
I20260924 05:01:43.813362 16245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:43.811360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41060 (local address 127.0.98.3:45917)
0924 05:01:43.811406 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 05:01:43.811408 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:01:43.811417 (+ 9us) server_negotiation.cc:407] Connection header received
0924 05:01:43.811471 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:01:43.811471 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:01:43.811487 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:01:43.811543 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:01:43.811645 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:43.812100 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:43.812788 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:01:43.812972 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:01:43.813013 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:01:43.813115 (+ 102us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:01:43.813208 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:01:43.813263 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 05:01:43.813276 (+ 13us) server_negotiation.cc:299] Negotiation successful
0924 05:01:43.813290 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 05:01:43.815006 16055 raft_consensus.cc:1261] T 2f7c2974b484419b8125d5a0d97d311b P 1d219c86d9ea4b6aa6d894eb20867698 [term 1 FOLLOWER]: Refusing update from remote peer e0cc0328540d4a9db537a475b14a4ab3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:01:43.815032 15921 raft_consensus.cc:1261] T 2f7c2974b484419b8125d5a0d97d311b P 6590438bffac4f5d8023294a6d2fd4ff [term 1 FOLLOWER]: Refusing update from remote peer e0cc0328540d4a9db537a475b14a4ab3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:01:43.815307 16257 consensus_queue.cc:1035] T 2f7c2974b484419b8125d5a0d97d311b P e0cc0328540d4a9db537a475b14a4ab3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d219c86d9ea4b6aa6d894eb20867698" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:01:43.815435 16259 consensus_queue.cc:1035] T 2f7c2974b484419b8125d5a0d97d311b P e0cc0328540d4a9db537a475b14a4ab3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6590438bffac4f5d8023294a6d2fd4ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:01:43.819943 15776 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46354:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:01:43.820026 15776 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46354:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:01:43.820659 15776 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e45cc94f6d2f434f9e8cf8dd78e8dff7: Sending DeleteTablet for 3 replicas of tablet 2f7c2974b484419b8125d5a0d97d311b
I20260924 05:01:43.821049 15901 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f7c2974b484419b8125d5a0d97d311b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:01:43 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33086
I20260924 05:01:43.821033 16169 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f7c2974b484419b8125d5a0d97d311b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:01:43 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41050
I20260924 05:01:43.821117 16035 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f7c2974b484419b8125d5a0d97d311b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:01:43 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38880
I20260924 05:01:43.821324 16273 tablet_replica.cc:330] T 2f7c2974b484419b8125d5a0d97d311b P 1d219c86d9ea4b6aa6d894eb20867698: stopping tablet replica
I20260924 05:01:43.821449 16273 raft_consensus.cc:2229] T 2f7c2974b484419b8125d5a0d97d311b P 1d219c86d9ea4b6aa6d894eb20867698 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:01:43.821480 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 15838
I20260924 05:01:43.821558 16273 raft_consensus.cc:2258] T 2f7c2974b484419b8125d5a0d97d311b P 1d219c86d9ea4b6aa6d894eb20867698 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260924 05:01:43.826252 15762 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33823 (error 108)
I20260924 05:01:43.826287 16272 tablet_replica.cc:330] T 2f7c2974b484419b8125d5a0d97d311b P e0cc0328540d4a9db537a475b14a4ab3: stopping tablet replica
I20260924 05:01:43.826480 16272 raft_consensus.cc:2229] T 2f7c2974b484419b8125d5a0d97d311b P e0cc0328540d4a9db537a475b14a4ab3 [term 1 LEADER]: Raft consensus shutting down.
I20260924 05:01:43.826606 16272 raft_consensus.cc:2258] T 2f7c2974b484419b8125d5a0d97d311b P e0cc0328540d4a9db537a475b14a4ab3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:01:43.827178 16244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:43.827037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33823 (local address 127.0.0.1:33096)
0924 05:01:43.827101 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:43.827142 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260924 05:01:43.827186 16273 ts_tablet_manager.cc:1905] T 2f7c2974b484419b8125d5a0d97d311b P 1d219c86d9ea4b6aa6d894eb20867698: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:01:43.827186 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 15972
W20260924 05:01:43.827692 15762 catalog_manager.cc:4584] TS 6590438bffac4f5d8023294a6d2fd4ff (127.0.98.1:33823): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2f7c2974b484419b8125d5a0d97d311b: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33823: connect: Connection refused (error 111)
I20260924 05:01:43.828588 16272 ts_tablet_manager.cc:1905] T 2f7c2974b484419b8125d5a0d97d311b P e0cc0328540d4a9db537a475b14a4ab3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:01:43.829988 16272 ts_tablet_manager.cc:1918] T 2f7c2974b484419b8125d5a0d97d311b P e0cc0328540d4a9db537a475b14a4ab3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 05:01:43.830060 16272 log.cc:1196] T 2f7c2974b484419b8125d5a0d97d311b P e0cc0328540d4a9db537a475b14a4ab3: Deleting WAL directory at /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-2/wal/wals/2f7c2974b484419b8125d5a0d97d311b
I20260924 05:01:43.830310 16272 ts_tablet_manager.cc:1939] T 2f7c2974b484419b8125d5a0d97d311b P e0cc0328540d4a9db537a475b14a4ab3: Deleting consensus metadata
I20260924 05:01:43.830722 15763 catalog_manager.cc:4841] TS e0cc0328540d4a9db537a475b14a4ab3 (127.0.98.3:45917): tablet 2f7c2974b484419b8125d5a0d97d311b (table test-table [id=677ab201edee4befae64df9fc951a4ef]) successfully deleted
I20260924 05:01:43.832811 16244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:01:43.832695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42831 (local address 127.0.0.1:38892)
0924 05:01:43.832751 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0924 05:01:43.832780 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:42831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260924 05:01:43.832870 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 16106
I20260924 05:01:43.837317 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 15744
I20260924 05:02:03.841727 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41625
--webserver_interface=127.0.98.62
--webserver_port=43227
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41625
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:02:03.920984 16275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:02:03.921195 16275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:02:03.921216 16275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:02:03.922364 16275 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 05:02:03.922405 16275 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 05:02:03.922416 16275 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 05:02:03.922426 16275 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:02:03.922438 16275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:02:03.922459 16275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:02:03.922482 16275 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:02:03.922497 16275 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:02:03.924005 16275 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41625
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=43227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16275
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:02:03.924230 16275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:02:03.924463 16275 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:02:03.932356 16282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:02:03.932359 16281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:02:03.932394 16275 server_base.cc:1034] running on GCE node
W20260924 05:02:03.932374 16284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:02:03.932813 16275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:02:03.933135 16275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:02:03.934301 16275 hybrid_clock.cc:648] HybridClock initialized: now 1790226123934282 us; error 35 us; skew 500 ppm
Sep 24 05:02:03 dist-test-slave-70st krb5kdc[15731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226123, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:02:03.936791 16275 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 05:02:03.937232 16275 webserver.cc:460] Webserver started at http://127.0.98.62:43227/ using document root <none> and password file <none>
I20260924 05:02:03.937486 16275 fs_manager.cc:359] Metadata directory not provided
I20260924 05:02:03.937547 16275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:02:03.938874 16275 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 05:02:03.939529 16291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:02:03.939690 16275 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:02:03.939781 16275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/master-0/wal
uuid: "e45cc94f6d2f434f9e8cf8dd78e8dff7"
format_stamp: "Formatted at 2026-09-24 05:01:43 on dist-test-slave-70st"
server_key: "bc285d33076a5410016734cc94a222b3"
server_key_iv: "9a02138c971c7442b414f0d78a1bcd86"
server_key_version: "encryptionkey@0"
I20260924 05:02:03.940178 16275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:02:03.958986 16275 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:02:03.960876 16275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:02:03.961015 16275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:02:03.964762 16275 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41625
I20260924 05:02:03.964820 16343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41625 every 8 connection(s)
I20260924 05:02:03.965183 16275 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:02:03.966583 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 16275
I20260924 05:02:03.966730 16344 sys_catalog.cc:263] Verifying existing consensus state
I20260924 05:02:03.966914 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:33823
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:41625
--webserver_port=44521
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 05:02:03.967613 16344 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e45cc94f6d2f434f9e8cf8dd78e8dff7: Bootstrap starting.
I20260924 05:02:03.975828 16344 log.cc:824] T 00000000000000000000000000000000 P e45cc94f6d2f434f9e8cf8dd78e8dff7: Log is configured to *not* fsync() on all Append() calls
I20260924 05:02:03.977943 16344 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e45cc94f6d2f434f9e8cf8dd78e8dff7: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 05:02:03.978214 16344 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e45cc94f6d2f434f9e8cf8dd78e8dff7: Bootstrap complete.
I20260924 05:02:03.980370 16344 raft_consensus.cc:348] T 00000000000000000000000000000000 P e45cc94f6d2f434f9e8cf8dd78e8dff7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e45cc94f6d2f434f9e8cf8dd78e8dff7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41625 } }
I20260924 05:02:03.980603 16344 raft_consensus.cc:729] T 00000000000000000000000000000000 P e45cc94f6d2f434f9e8cf8dd78e8dff7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e45cc94f6d2f434f9e8cf8dd78e8dff7, State: Initialized, Role: FOLLOWER
I20260924 05:02:03.980751 16344 consensus_queue.cc:260] T 00000000000000000000000000000000 P e45cc94f6d2f434f9e8cf8dd78e8dff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e45cc94f6d2f434f9e8cf8dd78e8dff7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41625 } }
I20260924 05:02:03.980818 16344 raft_consensus.cc:388] T 00000000000000000000000000000000 P e45cc94f6d2f434f9e8cf8dd78e8dff7 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:02:03.980852 16344 raft_consensus.cc:482] T 00000000000000000000000000000000 P e45cc94f6d2f434f9e8cf8dd78e8dff7 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:02:03.980899 16344 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e45cc94f6d2f434f9e8cf8dd78e8dff7 [term 1 FOLLOWER]: Advancing to term 2
I20260924 05:02:03.981987 16344 raft_consensus.cc:504] T 00000000000000000000000000000000 P e45cc94f6d2f434f9e8cf8dd78e8dff7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e45cc94f6d2f434f9e8cf8dd78e8dff7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41625 } }
I20260924 05:02:03.982082 16344 leader_election.cc:302] T 00000000000000000000000000000000 P e45cc94f6d2f434f9e8cf8dd78e8dff7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e45cc94f6d2f434f9e8cf8dd78e8dff7; no voters:
I20260924 05:02:03.982267 16344 leader_election.cc:288] T 00000000000000000000000000000000 P e45cc94f6d2f434f9e8cf8dd78e8dff7 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 05:02:03.982317 16348 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e45cc94f6d2f434f9e8cf8dd78e8dff7 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 05:02:03.982465 16348 raft_consensus.cc:686] T 00000000000000000000000000000000 P e45cc94f6d2f434f9e8cf8dd78e8dff7 [term 2 LEADER]: Becoming Leader. State: Replica: e45cc94f6d2f434f9e8cf8dd78e8dff7, State: Running, Role: LEADER
I20260924 05:02:03.982530 16344 sys_catalog.cc:564] T 00000000000000000000000000000000 P e45cc94f6d2f434f9e8cf8dd78e8dff7 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:02:03.982550 16348 consensus_queue.cc:237] T 00000000000000000000000000000000 P e45cc94f6d2f434f9e8cf8dd78e8dff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e45cc94f6d2f434f9e8cf8dd78e8dff7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41625 } }
I20260924 05:02:03.982968 16349 sys_catalog.cc:455] T 00000000000000000000000000000000 P e45cc94f6d2f434f9e8cf8dd78e8dff7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e45cc94f6d2f434f9e8cf8dd78e8dff7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e45cc94f6d2f434f9e8cf8dd78e8dff7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41625 } } }
I20260924 05:02:03.982987 16350 sys_catalog.cc:455] T 00000000000000000000000000000000 P e45cc94f6d2f434f9e8cf8dd78e8dff7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e45cc94f6d2f434f9e8cf8dd78e8dff7. Latest consensus state: current_term: 2 leader_uuid: "e45cc94f6d2f434f9e8cf8dd78e8dff7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e45cc94f6d2f434f9e8cf8dd78e8dff7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41625 } } }
I20260924 05:02:03.983199 16350 sys_catalog.cc:458] T 00000000000000000000000000000000 P e45cc94f6d2f434f9e8cf8dd78e8dff7 [sys.catalog]: This master's current role is: LEADER
I20260924 05:02:03.983372 16349 sys_catalog.cc:458] T 00000000000000000000000000000000 P e45cc94f6d2f434f9e8cf8dd78e8dff7 [sys.catalog]: This master's current role is: LEADER
I20260924 05:02:03.984189 16364 catalog_manager.cc:1259] Loaded cluster ID: 31fbd21556764b1e891939dae73550c7
I20260924 05:02:03.984226 16364 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e45cc94f6d2f434f9e8cf8dd78e8dff7: loading cluster ID for follower catalog manager: success
I20260924 05:02:03.985013 16364 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e45cc94f6d2f434f9e8cf8dd78e8dff7: acquiring CA information for follower catalog manager: success
I20260924 05:02:03.985348 16364 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e45cc94f6d2f434f9e8cf8dd78e8dff7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 05:02:03.985435 16358 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:02:03.985760 16358 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:02:03.985829 16358 catalog_manager.cc:1259] Loaded cluster ID: 31fbd21556764b1e891939dae73550c7
I20260924 05:02:03.985882 16358 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:02:03.986043 16358 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:02:03.986131 16358 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e45cc94f6d2f434f9e8cf8dd78e8dff7: Loaded TSK: 0
I20260924 05:02:03.996312 16358 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e45cc94f6d2f434f9e8cf8dd78e8dff7: Generated new TSK 1
I20260924 05:02:03.996399 16358 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 05:02:04.063740 16346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:02:04.063956 16346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:02:04.063982 16346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:02:04.065172 16346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:02:04.065215 16346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:02:04.065241 16346 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 05:02:04.066756 16346 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:33823
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=44521
--tserver_master_addrs=127.0.98.62:41625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16346
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:02:04.066975 16346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:02:04.067234 16346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:02:04.072132 16371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:02:04.072151 16372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:02:04.072160 16374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:02:04.072422 16346 server_base.cc:1034] running on GCE node
I20260924 05:02:04.072613 16346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:02:04.072849 16346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:02:04.073997 16346 hybrid_clock.cc:648] HybridClock initialized: now 1790226124073984 us; error 33 us; skew 500 ppm
Sep 24 05:02:04 dist-test-slave-70st krb5kdc[15731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226124, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:02:04.076365 16346 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 05:02:04.076802 16346 webserver.cc:460] Webserver started at http://127.0.98.1:44521/ using document root <none> and password file <none>
I20260924 05:02:04.077021 16346 fs_manager.cc:359] Metadata directory not provided
I20260924 05:02:04.077081 16346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:02:04.078516 16346 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 05:02:04.079180 16381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:02:04.079344 16346 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:02:04.079424 16346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-0/wal
uuid: "6590438bffac4f5d8023294a6d2fd4ff"
format_stamp: "Formatted at 2026-09-24 05:01:43 on dist-test-slave-70st"
server_key: "b4f01fdb4ff7c2a227ab16508d224509"
server_key_iv: "359943cd0bcef6f5b6d2416d8cd8b768"
server_key_version: "encryptionkey@0"
I20260924 05:02:04.079783 16346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:02:04.099314 16346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:02:04.101459 16346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:02:04.101603 16346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:02:04.101836 16346 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:02:04.102291 16388 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 05:02:04.104168 16346 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 05:02:04.104223 16346 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260924 05:02:04.104256 16346 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 05:02:04.104871 16346 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 05:02:04.104923 16346 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260924 05:02:04.104998 16388 tablet_bootstrap.cc:492] T 2f7c2974b484419b8125d5a0d97d311b P 6590438bffac4f5d8023294a6d2fd4ff: Bootstrap starting.
I20260924 05:02:04.111675 16346 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33823
I20260924 05:02:04.112092 16346 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-0/data/info.pb
I20260924 05:02:04.114104 16495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33823 every 8 connection(s)
I20260924 05:02:04.116058 16388 log.cc:824] T 2f7c2974b484419b8125d5a0d97d311b P 6590438bffac4f5d8023294a6d2fd4ff: Log is configured to *not* fsync() on all Append() calls
I20260924 05:02:04.118682 16388 tablet_bootstrap.cc:492] T 2f7c2974b484419b8125d5a0d97d311b P 6590438bffac4f5d8023294a6d2fd4ff: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 05:02:04.118989 16388 tablet_bootstrap.cc:492] T 2f7c2974b484419b8125d5a0d97d311b P 6590438bffac4f5d8023294a6d2fd4ff: Bootstrap complete.
Sep 24 05:02:04 dist-test-slave-70st krb5kdc[15731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226124, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:02:04.120040 16388 ts_tablet_manager.cc:1397] T 2f7c2974b484419b8125d5a0d97d311b P 6590438bffac4f5d8023294a6d2fd4ff: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.011s
I20260924 05:02:04.121287 16388 raft_consensus.cc:348] T 2f7c2974b484419b8125d5a0d97d311b P 6590438bffac4f5d8023294a6d2fd4ff [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cc0328540d4a9db537a475b14a4ab3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45917 } } peers { permanent_uuid: "6590438bffac4f5d8023294a6d2fd4ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33823 } } peers { permanent_uuid: "1d219c86d9ea4b6aa6d894eb20867698" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42831 } }
I20260924 05:02:04.121681 16388 raft_consensus.cc:729] T 2f7c2974b484419b8125d5a0d97d311b P 6590438bffac4f5d8023294a6d2fd4ff [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6590438bffac4f5d8023294a6d2fd4ff, State: Initialized, Role: FOLLOWER
I20260924 05:02:04.121838 16388 consensus_queue.cc:260] T 2f7c2974b484419b8125d5a0d97d311b P 6590438bffac4f5d8023294a6d2fd4ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cc0328540d4a9db537a475b14a4ab3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45917 } } peers { permanent_uuid: "6590438bffac4f5d8023294a6d2fd4ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33823 } } peers { permanent_uuid: "1d219c86d9ea4b6aa6d894eb20867698" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42831 } }
I20260924 05:02:04.122119 16388 ts_tablet_manager.cc:1428] T 2f7c2974b484419b8125d5a0d97d311b P 6590438bffac4f5d8023294a6d2fd4ff: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20260924 05:02:04.122243 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 16346
I20260924 05:02:04.122502 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:42831
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:41625
--webserver_port=44059
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 05:02:04.124993 16499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:04.112882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51115 (local address 127.0.98.62:41625)
0924 05:02:04.113560 (+ 678us) server_negotiation.cc:206] Beginning negotiation
0924 05:02:04.113569 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 05:02:04.113609 (+ 40us) server_negotiation.cc:407] Connection header received
0924 05:02:04.113750 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:02:04.113754 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:02:04.113798 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:02:04.113914 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:02:04.114684 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:04.115874 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:04.117242 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:04.117702 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:04.117804 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:04.119931 (+ 2127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:02:04.119935 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:02:04.119935 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:02:04.119947 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:02:04.121520 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:02:04.121965 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:02:04.121968 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:02:04.121969 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:02:04.123661 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:02:04.124023 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:02:04.124024 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:02:04.124024 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:02:04.124324 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:02:04.124409 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 05:02:04.124748 (+ 339us) server_negotiation.cc:299] Negotiation successful
0924 05:02:04.124851 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":622,"thread_start_us":90,"threads_started":1}
I20260924 05:02:04.125149 16498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:04.112760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41625 (local address 127.0.98.1:51115)
0924 05:02:04.112935 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:04.112983 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 05:02:04.113523 (+ 540us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:02:04.113954 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:02:04.113958 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:02:04.114337 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:02:04.114566 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:04.114572 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:04.116022 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:04.116024 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:04.117148 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:04.117151 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:04.117808 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:04.117809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:04.117935 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:04.118432 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:02:04.118440 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:02:04.119812 (+ 1372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:02:04.121586 (+ 1774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:02:04.121593 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:02:04.121595 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:02:04.121881 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:02:04.123773 (+ 1892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:02:04.123774 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:02:04.123775 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:02:04.123933 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:02:04.124403 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:02:04.124409 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:02:04.124576 (+ 167us) client_negotiation.cc:769] Sending connection context
0924 05:02:04.124695 (+ 119us) client_negotiation.cc:240] Negotiation successful
0924 05:02:04.124875 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":88,"threads_started":1}
I20260924 05:02:04.125751 16496 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41625
I20260924 05:02:04.125841 16496 heartbeater.cc:464] Registering TS with master...
I20260924 05:02:04.126005 16496 heartbeater.cc:511] Master 127.0.98.62:41625 requested a full tablet report, sending...
I20260924 05:02:04.126643 16308 ts_manager.cc:194] Registered new tserver with Master: 6590438bffac4f5d8023294a6d2fd4ff (127.0.98.1:33823)
I20260924 05:02:04.127734 16308 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:51115
I20260924 05:02:04.128633 16496 heartbeater.cc:503] Master 127.0.98.62:41625 was elected leader, sending a full tablet report...
Sep 24 05:02:04 dist-test-slave-70st krb5kdc[15731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226123, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 05:02:04.134889 16503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:04.127178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33823 (local address 127.0.0.1:38212)
0924 05:02:04.127585 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:04.127617 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 05:02:04.127679 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:02:04.127927 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:02:04.127931 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:02:04.128418 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:02:04.128474 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:04.128477 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:04.129490 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:04.129491 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:04.130041 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:04.130043 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:04.130466 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:04.130470 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:04.130539 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:04.130976 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:02:04.130980 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:02:04.132258 (+ 1278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:02:04.133870 (+ 1612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:02:04.133877 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:02:04.133879 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:02:04.134061 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:02:04.134245 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:02:04.134245 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:02:04.134246 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:02:04.134300 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:02:04.134539 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:02:04.134545 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:02:04.134596 (+ 51us) client_negotiation.cc:769] Sending connection context
0924 05:02:04.134668 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 05:02:04.134785 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":84,"threads_started":1}
I20260924 05:02:04.134889 16504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:04.127390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38212 (local address 127.0.98.1:33823)
0924 05:02:04.127529 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 05:02:04.127534 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 05:02:04.127762 (+ 228us) server_negotiation.cc:407] Connection header received
0924 05:02:04.127797 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:02:04.127799 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:02:04.127816 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:02:04.127871 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:02:04.128562 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:04.129399 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:04.130114 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:04.130405 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:04.130462 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:04.132348 (+ 1886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:02:04.132356 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:02:04.132356 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:02:04.132362 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:02:04.133797 (+ 1435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:02:04.134130 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:02:04.134134 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:02:04.134134 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:02:04.134185 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:02:04.134348 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:02:04.134349 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:02:04.134349 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:02:04.134482 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:02:04.134550 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 05:02:04.134707 (+ 157us) server_negotiation.cc:299] Negotiation successful
0924 05:02:04.134797 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":59,"threads_started":1}
I20260924 05:02:04.135319 16430 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f7c2974b484419b8125d5a0d97d311b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:01:43 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38212
I20260924 05:02:04.135794 16505 tablet_replica.cc:330] stopping tablet replica
I20260924 05:02:04.135887 16505 raft_consensus.cc:2229] T 2f7c2974b484419b8125d5a0d97d311b P 6590438bffac4f5d8023294a6d2fd4ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:02:04.135957 16505 raft_consensus.cc:2258] T 2f7c2974b484419b8125d5a0d97d311b P 6590438bffac4f5d8023294a6d2fd4ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:02:04.136235 16505 ts_tablet_manager.cc:1905] T 2f7c2974b484419b8125d5a0d97d311b P 6590438bffac4f5d8023294a6d2fd4ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:02:04.138357 16505 ts_tablet_manager.cc:1918] T 2f7c2974b484419b8125d5a0d97d311b P 6590438bffac4f5d8023294a6d2fd4ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 05:02:04.138417 16505 log.cc:1196] T 2f7c2974b484419b8125d5a0d97d311b P 6590438bffac4f5d8023294a6d2fd4ff: Deleting WAL directory at /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-0/wal/wals/2f7c2974b484419b8125d5a0d97d311b
I20260924 05:02:04.138597 16505 ts_tablet_manager.cc:1939] T 2f7c2974b484419b8125d5a0d97d311b P 6590438bffac4f5d8023294a6d2fd4ff: Deleting consensus metadata
I20260924 05:02:04.139065 16294 catalog_manager.cc:4841] TS 6590438bffac4f5d8023294a6d2fd4ff (127.0.98.1:33823): tablet 2f7c2974b484419b8125d5a0d97d311b (table test-table [id=677ab201edee4befae64df9fc951a4ef]) successfully deleted
W20260924 05:02:04.205112 16502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:02:04.205322 16502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:02:04.205363 16502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:02:04.206539 16502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:02:04.206581 16502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:02:04.206615 16502 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 05:02:04.208052 16502 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:42831
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=44059
--tserver_master_addrs=127.0.98.62:41625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16502
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:02:04.208292 16502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:02:04.208504 16502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:02:04.213164 16511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:02:04.213177 16510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:02:04.213178 16513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:02:04.213516 16502 server_base.cc:1034] running on GCE node
I20260924 05:02:04.213708 16502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:02:04.213940 16502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:02:04.215082 16502 hybrid_clock.cc:648] HybridClock initialized: now 1790226124215066 us; error 27 us; skew 500 ppm
Sep 24 05:02:04 dist-test-slave-70st krb5kdc[15731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226124, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:02:04.217408 16502 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 05:02:04.217833 16502 webserver.cc:460] Webserver started at http://127.0.98.2:44059/ using document root <none> and password file <none>
I20260924 05:02:04.218027 16502 fs_manager.cc:359] Metadata directory not provided
I20260924 05:02:04.218096 16502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:02:04.219349 16502 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 05:02:04.219949 16520 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:02:04.220103 16502 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:02:04.220180 16502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-1/wal
uuid: "1d219c86d9ea4b6aa6d894eb20867698"
format_stamp: "Formatted at 2026-09-24 05:01:43 on dist-test-slave-70st"
server_key: "9182aeba844e056977f6da6dfdfdbd18"
server_key_iv: "0cc71887ec5726087abc5e4c97c68e26"
server_key_version: "encryptionkey@0"
I20260924 05:02:04.220505 16502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:02:04.238093 16502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:02:04.239881 16502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:02:04.240001 16502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:02:04.240197 16502 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:02:04.240646 16527 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 05:02:04.242110 16502 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 05:02:04.242152 16502 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260924 05:02:04.242177 16502 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 05:02:04.242775 16502 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 05:02:04.242805 16502 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260924 05:02:04.242905 16527 tablet_bootstrap.cc:492] T 2f7c2974b484419b8125d5a0d97d311b P 1d219c86d9ea4b6aa6d894eb20867698: Bootstrap starting.
I20260924 05:02:04.244520 16527 log.cc:824] T 2f7c2974b484419b8125d5a0d97d311b P 1d219c86d9ea4b6aa6d894eb20867698: Log is configured to *not* fsync() on all Append() calls
I20260924 05:02:04.247005 16527 tablet_bootstrap.cc:492] T 2f7c2974b484419b8125d5a0d97d311b P 1d219c86d9ea4b6aa6d894eb20867698: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 05:02:04.247277 16527 tablet_bootstrap.cc:492] T 2f7c2974b484419b8125d5a0d97d311b P 1d219c86d9ea4b6aa6d894eb20867698: Bootstrap complete.
I20260924 05:02:04.247421 16527 ts_tablet_manager.cc:1397] T 2f7c2974b484419b8125d5a0d97d311b P 1d219c86d9ea4b6aa6d894eb20867698: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20260924 05:02:04.248855 16502 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42831
I20260924 05:02:04.248888 16635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42831 every 8 connection(s)
I20260924 05:02:04.248860 16527 raft_consensus.cc:348] T 2f7c2974b484419b8125d5a0d97d311b P 1d219c86d9ea4b6aa6d894eb20867698 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cc0328540d4a9db537a475b14a4ab3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45917 } } peers { permanent_uuid: "6590438bffac4f5d8023294a6d2fd4ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33823 } } peers { permanent_uuid: "1d219c86d9ea4b6aa6d894eb20867698" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42831 } }
I20260924 05:02:04.249184 16502 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-1/data/info.pb
I20260924 05:02:04.249220 16527 raft_consensus.cc:729] T 2f7c2974b484419b8125d5a0d97d311b P 1d219c86d9ea4b6aa6d894eb20867698 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d219c86d9ea4b6aa6d894eb20867698, State: Initialized, Role: FOLLOWER
I20260924 05:02:04.249378 16527 consensus_queue.cc:260] T 2f7c2974b484419b8125d5a0d97d311b P 1d219c86d9ea4b6aa6d894eb20867698 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cc0328540d4a9db537a475b14a4ab3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45917 } } peers { permanent_uuid: "6590438bffac4f5d8023294a6d2fd4ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33823 } } peers { permanent_uuid: "1d219c86d9ea4b6aa6d894eb20867698" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42831 } }
I20260924 05:02:04.249826 16527 ts_tablet_manager.cc:1428] T 2f7c2974b484419b8125d5a0d97d311b P 1d219c86d9ea4b6aa6d894eb20867698: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
Sep 24 05:02:04 dist-test-slave-70st krb5kdc[15731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226124, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:02:04.257360 16499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:04.249896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34783 (local address 127.0.98.62:41625)
0924 05:02:04.249954 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 05:02:04.249957 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:02:04.250125 (+ 168us) server_negotiation.cc:407] Connection header received
0924 05:02:04.250266 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:02:04.250267 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:02:04.250279 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:02:04.250336 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:02:04.250938 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:04.251439 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:04.252408 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:04.252614 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:04.252668 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:04.255151 (+ 2483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:02:04.255152 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:02:04.255153 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:02:04.255161 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:02:04.256291 (+ 1130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:02:04.256647 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:02:04.256647 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:02:04.256648 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:02:04.256693 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:02:04.256872 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:02:04.256872 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:02:04.256872 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:02:04.256970 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:02:04.257015 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 05:02:04.257223 (+ 208us) server_negotiation.cc:299] Negotiation successful
0924 05:02:04.257289 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 05:02:04.257551 16639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:04.249865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41625 (local address 127.0.98.2:34783)
0924 05:02:04.250006 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:04.250045 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 05:02:04.250193 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:02:04.250393 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:02:04.250397 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:02:04.250658 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:02:04.250845 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:04.250849 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:04.251537 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:04.251538 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:04.252319 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:04.252321 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:04.252680 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:04.252681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:04.252815 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:04.253277 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:02:04.253285 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:02:04.255065 (+ 1780us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:02:04.256357 (+ 1292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:02:04.256363 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:02:04.256364 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:02:04.256590 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:02:04.256736 (+ 146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:02:04.256737 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:02:04.256737 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:02:04.256832 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:02:04.257012 (+ 180us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:02:04.257015 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:02:04.257131 (+ 116us) client_negotiation.cc:769] Sending connection context
0924 05:02:04.257207 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 05:02:04.257328 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":73,"threads_started":1}
I20260924 05:02:04.258019 16636 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41625
I20260924 05:02:04.258114 16636 heartbeater.cc:464] Registering TS with master...
I20260924 05:02:04.258280 16636 heartbeater.cc:511] Master 127.0.98.62:41625 requested a full tablet report, sending...
I20260924 05:02:04.258705 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 16502
I20260924 05:02:04.258786 16308 ts_manager.cc:194] Registered new tserver with Master: 1d219c86d9ea4b6aa6d894eb20867698 (127.0.98.2:42831)
I20260924 05:02:04.258992 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:45917
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:41625
--webserver_port=40245
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 05:02:04.259361 16308 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:34783
I20260924 05:02:04.260175 16636 heartbeater.cc:503] Master 127.0.98.62:41625 was elected leader, sending a full tablet report...
Sep 24 05:02:04 dist-test-slave-70st krb5kdc[15731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226123, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260924 05:02:04.266479 16503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:04.259090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42831 (local address 127.0.0.1:56286)
0924 05:02:04.259488 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:04.259498 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:02:04.259547 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:02:04.259704 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:02:04.259705 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:02:04.259847 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:02:04.259886 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:04.259888 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:04.260999 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:04.261000 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:04.261531 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:04.261533 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:04.262097 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:04.262099 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:04.262147 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:04.262555 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:02:04.262561 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:02:04.263926 (+ 1365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:02:04.265498 (+ 1572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:02:04.265500 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:02:04.265500 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:02:04.265662 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:02:04.265840 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:02:04.265841 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:02:04.265841 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:02:04.265898 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:02:04.266162 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:02:04.266163 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:02:04.266209 (+ 46us) client_negotiation.cc:769] Sending connection context
0924 05:02:04.266271 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 05:02:04.266377 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382}
I20260924 05:02:04.266500 16640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:04.259195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56286 (local address 127.0.98.2:42831)
0924 05:02:04.259351 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 05:02:04.259356 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 05:02:04.259594 (+ 238us) server_negotiation.cc:407] Connection header received
0924 05:02:04.259631 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:02:04.259633 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:02:04.259648 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:02:04.259704 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:02:04.259952 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:04.260905 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:04.261599 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:04.262033 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:04.262103 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:04.264006 (+ 1903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:02:04.264012 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:02:04.264013 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:02:04.264018 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:02:04.265406 (+ 1388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:02:04.265728 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:02:04.265732 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:02:04.265732 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:02:04.265786 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:02:04.265961 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:02:04.265962 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:02:04.265962 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:02:04.266102 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:02:04.266170 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 05:02:04.266305 (+ 135us) server_negotiation.cc:299] Negotiation successful
0924 05:02:04.266388 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":85,"threads_started":1}
I20260924 05:02:04.266959 16570 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f7c2974b484419b8125d5a0d97d311b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:01:43 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56286
I20260924 05:02:04.267524 16642 tablet_replica.cc:330] stopping tablet replica
I20260924 05:02:04.267632 16642 raft_consensus.cc:2229] T 2f7c2974b484419b8125d5a0d97d311b P 1d219c86d9ea4b6aa6d894eb20867698 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:02:04.267701 16642 raft_consensus.cc:2258] T 2f7c2974b484419b8125d5a0d97d311b P 1d219c86d9ea4b6aa6d894eb20867698 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:02:04.267997 16642 ts_tablet_manager.cc:1905] T 2f7c2974b484419b8125d5a0d97d311b P 1d219c86d9ea4b6aa6d894eb20867698: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:02:04.270207 16642 ts_tablet_manager.cc:1918] T 2f7c2974b484419b8125d5a0d97d311b P 1d219c86d9ea4b6aa6d894eb20867698: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 05:02:04.270270 16642 log.cc:1196] T 2f7c2974b484419b8125d5a0d97d311b P 1d219c86d9ea4b6aa6d894eb20867698: Deleting WAL directory at /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-1/wal/wals/2f7c2974b484419b8125d5a0d97d311b
I20260924 05:02:04.270455 16642 ts_tablet_manager.cc:1939] T 2f7c2974b484419b8125d5a0d97d311b P 1d219c86d9ea4b6aa6d894eb20867698: Deleting consensus metadata
I20260924 05:02:04.270901 16294 catalog_manager.cc:4841] TS 1d219c86d9ea4b6aa6d894eb20867698 (127.0.98.2:42831): tablet 2f7c2974b484419b8125d5a0d97d311b (table test-table [id=677ab201edee4befae64df9fc951a4ef]) successfully deleted
W20260924 05:02:04.339231 16641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:02:04.339448 16641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:02:04.339488 16641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:02:04.340682 16641 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:02:04.340723 16641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:02:04.340758 16641 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 05:02:04.342305 16641 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:45917
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=40245
--tserver_master_addrs=127.0.98.62:41625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16641
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:02:04.342551 16641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:02:04.342767 16641 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:02:04.347620 16648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:02:04.347607 16650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:02:04.347608 16647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:02:04.347654 16641 server_base.cc:1034] running on GCE node
I20260924 05:02:04.347954 16641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:02:04.348151 16641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:02:04.349282 16641 hybrid_clock.cc:648] HybridClock initialized: now 1790226124349269 us; error 29 us; skew 500 ppm
Sep 24 05:02:04 dist-test-slave-70st krb5kdc[15731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226124, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:02:04.351569 16641 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 05:02:04.351979 16641 webserver.cc:460] Webserver started at http://127.0.98.3:40245/ using document root <none> and password file <none>
I20260924 05:02:04.352180 16641 fs_manager.cc:359] Metadata directory not provided
I20260924 05:02:04.352232 16641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:02:04.353582 16641 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 05:02:04.354115 16657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:02:04.354241 16641 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 05:02:04.354313 16641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-2/wal
uuid: "e0cc0328540d4a9db537a475b14a4ab3"
format_stamp: "Formatted at 2026-09-24 05:01:43 on dist-test-slave-70st"
server_key: "8e0ce449eaf3849a57093216055d4807"
server_key_iv: "c4d89a7ff885b5cdf2cb5321864d74dc"
server_key_version: "encryptionkey@0"
I20260924 05:02:04.354651 16641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:02:04.373553 16641 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:02:04.375777 16641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:02:04.375913 16641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:02:04.376122 16641 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:02:04.376407 16641 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:02:04.376441 16641 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:02:04.376470 16641 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:02:04.376493 16641 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:02:04.381968 16641 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45917
I20260924 05:02:04.382022 16770 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45917 every 8 connection(s)
I20260924 05:02:04.382360 16641 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-2/data/info.pb
I20260924 05:02:04.384187 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 16641
Sep 24 05:02:04 dist-test-slave-70st krb5kdc[15731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226124, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:02:04.390697 16775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:04.384570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35414 (local address 127.0.98.62:41625)
0924 05:02:04.384695 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0924 05:02:04.384697 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:02:04.384770 (+ 73us) server_negotiation.cc:407] Connection header received
0924 05:02:04.384822 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:02:04.384823 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:02:04.384833 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:02:04.384901 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:02:04.385558 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:04.386125 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:04.386853 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:04.387081 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:04.387134 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:04.388301 (+ 1167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:02:04.388302 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:02:04.388303 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:02:04.388310 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:02:04.389681 (+ 1371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:02:04.389982 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:02:04.389983 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:02:04.389984 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:02:04.390032 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:02:04.390185 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:02:04.390186 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:02:04.390186 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:02:04.390325 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:02:04.390458 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0924 05:02:04.390520 (+ 62us) server_negotiation.cc:299] Negotiation successful
0924 05:02:04.390597 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":105,"thread_start_us":69,"threads_started":1}
I20260924 05:02:04.391840 16499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:04.383096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53267 (local address 127.0.98.62:41625)
0924 05:02:04.383146 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 05:02:04.383148 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:02:04.383449 (+ 301us) server_negotiation.cc:407] Connection header received
0924 05:02:04.383920 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:02:04.383921 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:02:04.383931 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:02:04.383981 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:02:04.385135 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:04.385812 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:04.386870 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:04.387076 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:04.387131 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:04.389324 (+ 2193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:02:04.389325 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:02:04.389325 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:02:04.389331 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:02:04.390554 (+ 1223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:02:04.390997 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:02:04.390997 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:02:04.390998 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:02:04.391049 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:02:04.391267 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:02:04.391268 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:02:04.391268 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:02:04.391367 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:02:04.391419 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 05:02:04.391681 (+ 262us) server_negotiation.cc:299] Negotiation successful
0924 05:02:04.391754 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":19}
I20260924 05:02:04.392032 16773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:04.383140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41625 (local address 127.0.98.3:53267)
0924 05:02:04.383317 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:04.383364 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 05:02:04.383831 (+ 467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:02:04.384055 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:02:04.384059 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:02:04.384782 (+ 723us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:02:04.385048 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:04.385053 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:04.385883 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:04.385884 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:04.386790 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:04.386793 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:04.387144 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:04.387145 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:04.387276 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:04.387786 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:02:04.387794 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:02:04.389233 (+ 1439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:02:04.390628 (+ 1395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:02:04.390636 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:02:04.390638 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:02:04.390937 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:02:04.391102 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:02:04.391103 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:02:04.391103 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:02:04.391211 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:02:04.391424 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:02:04.391430 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:02:04.391565 (+ 135us) client_negotiation.cc:769] Sending connection context
0924 05:02:04.391653 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 05:02:04.391815 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":88,"threads_started":1}
I20260924 05:02:04.392407 16771 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41625
I20260924 05:02:04.392504 16771 heartbeater.cc:464] Registering TS with master...
I20260924 05:02:04.392673 16771 heartbeater.cc:511] Master 127.0.98.62:41625 requested a full tablet report, sending...
I20260924 05:02:04.393010 16308 ts_manager.cc:194] Registered new tserver with Master: e0cc0328540d4a9db537a475b14a4ab3 (127.0.98.3:45917)
I20260924 05:02:04.393355 16308 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:53267
I20260924 05:02:04.402128 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 05:02:04.404978 16499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:04.402479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35424 (local address 127.0.98.62:41625)
0924 05:02:04.402553 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0924 05:02:04.402556 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:02:04.402647 (+ 91us) server_negotiation.cc:407] Connection header received
0924 05:02:04.402756 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:02:04.402757 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:02:04.402798 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:02:04.402858 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:02:04.403007 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:04.403529 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:04.404311 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:04.404497 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:04.404558 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:04.404724 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:02:04.404770 (+ 46us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0924 05:02:04.404907 (+ 137us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35424: token expired
Metrics: {"server-negotiator.queue_time_us":37}
W20260924 05:02:04.405048 16499 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35424: token expired
W20260924 05:02:04.405099 16239 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.98.62:41625)
I20260924 05:02:04.410262 16499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:04.405344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35434 (local address 127.0.98.62:41625)
0924 05:02:04.405385 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 05:02:04.405387 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:02:04.405395 (+ 8us) server_negotiation.cc:407] Connection header received
0924 05:02:04.405412 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:02:04.405433 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:02:04.405442 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:02:04.405485 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:02:04.405937 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:04.406369 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:04.407018 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:04.407205 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:04.407255 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:04.408279 (+ 1024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:02:04.408280 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:02:04.408281 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:02:04.408287 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:02:04.409271 (+ 984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:02:04.409611 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:02:04.409612 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:02:04.409613 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:02:04.409665 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:02:04.409819 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:02:04.409820 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:02:04.409820 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:02:04.409930 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:02:04.410016 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 05:02:04.410107 (+ 91us) server_negotiation.cc:299] Negotiation successful
0924 05:02:04.410177 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 05:02:04.411206 16308 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35434:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 05:02:04.412792 16308 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 05:02:04.414680 16570 tablet_service.cc:1467] Processing CreateTablet for tablet 4ba7192adfca4e73bb9cca620676f6cf (DEFAULT_TABLE table=test-table [id=19f1f35da8324795a915fc7e6cc51e97]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:02:04.414685 16430 tablet_service.cc:1467] Processing CreateTablet for tablet 4ba7192adfca4e73bb9cca620676f6cf (DEFAULT_TABLE table=test-table [id=19f1f35da8324795a915fc7e6cc51e97]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:02:04.414850 16570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ba7192adfca4e73bb9cca620676f6cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:02:04.414865 16430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ba7192adfca4e73bb9cca620676f6cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:02:04.416152 16527 tablet_bootstrap.cc:492] T 4ba7192adfca4e73bb9cca620676f6cf P 1d219c86d9ea4b6aa6d894eb20867698: Bootstrap starting.
I20260924 05:02:04.416324 16388 tablet_bootstrap.cc:492] T 4ba7192adfca4e73bb9cca620676f6cf P 6590438bffac4f5d8023294a6d2fd4ff: Bootstrap starting.
I20260924 05:02:04.416549 16527 tablet_bootstrap.cc:654] T 4ba7192adfca4e73bb9cca620676f6cf P 1d219c86d9ea4b6aa6d894eb20867698: Neither blocks nor log segments found. Creating new log.
I20260924 05:02:04.416575 16388 tablet_bootstrap.cc:654] T 4ba7192adfca4e73bb9cca620676f6cf P 6590438bffac4f5d8023294a6d2fd4ff: Neither blocks nor log segments found. Creating new log.
I20260924 05:02:04.417348 16527 tablet_bootstrap.cc:492] T 4ba7192adfca4e73bb9cca620676f6cf P 1d219c86d9ea4b6aa6d894eb20867698: No bootstrap required, opened a new log
I20260924 05:02:04.417390 16527 ts_tablet_manager.cc:1397] T 4ba7192adfca4e73bb9cca620676f6cf P 1d219c86d9ea4b6aa6d894eb20867698: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 05:02:04.417546 16388 tablet_bootstrap.cc:492] T 4ba7192adfca4e73bb9cca620676f6cf P 6590438bffac4f5d8023294a6d2fd4ff: No bootstrap required, opened a new log
I20260924 05:02:04.417557 16527 raft_consensus.cc:348] T 4ba7192adfca4e73bb9cca620676f6cf P 1d219c86d9ea4b6aa6d894eb20867698 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cc0328540d4a9db537a475b14a4ab3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45917 } } peers { permanent_uuid: "6590438bffac4f5d8023294a6d2fd4ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33823 } } peers { permanent_uuid: "1d219c86d9ea4b6aa6d894eb20867698" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42831 } }
I20260924 05:02:04.417599 16388 ts_tablet_manager.cc:1397] T 4ba7192adfca4e73bb9cca620676f6cf P 6590438bffac4f5d8023294a6d2fd4ff: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260924 05:02:04.417618 16527 raft_consensus.cc:374] T 4ba7192adfca4e73bb9cca620676f6cf P 1d219c86d9ea4b6aa6d894eb20867698 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:02:04.417660 16527 raft_consensus.cc:729] T 4ba7192adfca4e73bb9cca620676f6cf P 1d219c86d9ea4b6aa6d894eb20867698 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d219c86d9ea4b6aa6d894eb20867698, State: Initialized, Role: FOLLOWER
I20260924 05:02:04.417702 16527 consensus_queue.cc:260] T 4ba7192adfca4e73bb9cca620676f6cf P 1d219c86d9ea4b6aa6d894eb20867698 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cc0328540d4a9db537a475b14a4ab3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45917 } } peers { permanent_uuid: "6590438bffac4f5d8023294a6d2fd4ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33823 } } peers { permanent_uuid: "1d219c86d9ea4b6aa6d894eb20867698" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42831 } }
I20260924 05:02:04.417738 16388 raft_consensus.cc:348] T 4ba7192adfca4e73bb9cca620676f6cf P 6590438bffac4f5d8023294a6d2fd4ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cc0328540d4a9db537a475b14a4ab3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45917 } } peers { permanent_uuid: "6590438bffac4f5d8023294a6d2fd4ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33823 } } peers { permanent_uuid: "1d219c86d9ea4b6aa6d894eb20867698" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42831 } }
I20260924 05:02:04.417793 16527 ts_tablet_manager.cc:1428] T 4ba7192adfca4e73bb9cca620676f6cf P 1d219c86d9ea4b6aa6d894eb20867698: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 05:02:04.417805 16388 raft_consensus.cc:374] T 4ba7192adfca4e73bb9cca620676f6cf P 6590438bffac4f5d8023294a6d2fd4ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:02:04.417860 16388 raft_consensus.cc:729] T 4ba7192adfca4e73bb9cca620676f6cf P 6590438bffac4f5d8023294a6d2fd4ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6590438bffac4f5d8023294a6d2fd4ff, State: Initialized, Role: FOLLOWER
I20260924 05:02:04.417919 16388 consensus_queue.cc:260] T 4ba7192adfca4e73bb9cca620676f6cf P 6590438bffac4f5d8023294a6d2fd4ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cc0328540d4a9db537a475b14a4ab3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45917 } } peers { permanent_uuid: "6590438bffac4f5d8023294a6d2fd4ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33823 } } peers { permanent_uuid: "1d219c86d9ea4b6aa6d894eb20867698" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42831 } }
I20260924 05:02:04.418017 16388 ts_tablet_manager.cc:1428] T 4ba7192adfca4e73bb9cca620676f6cf P 6590438bffac4f5d8023294a6d2fd4ff: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 05:02:04.418937 16503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:04.414458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45917 (local address 127.0.0.1:55698)
0924 05:02:04.414504 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:04.414513 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:02:04.414550 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:02:04.415041 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:02:04.415042 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:02:04.415044 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:02:04.415080 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:04.415082 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:04.416351 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:04.416352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:04.417201 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:04.417204 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:04.418732 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:04.418732 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:04.418794 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:04.418809 (+ 15us) client_negotiation.cc:769] Sending connection context
0924 05:02:04.418852 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 05:02:04.418882 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260924 05:02:04.418960 16777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:04.414635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55698 (local address 127.0.98.3:45917)
0924 05:02:04.414858 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0924 05:02:04.414863 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 05:02:04.414874 (+ 11us) server_negotiation.cc:407] Connection header received
0924 05:02:04.414913 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:02:04.414916 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:02:04.414932 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:02:04.415032 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:02:04.415135 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:04.416249 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:04.417300 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:04.418662 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:04.418731 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:04.418783 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 05:02:04.418865 (+ 82us) server_negotiation.cc:299] Negotiation successful
0924 05:02:04.418895 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":113,"threads_started":1}
I20260924 05:02:04.419551 16705 tablet_service.cc:1467] Processing CreateTablet for tablet 4ba7192adfca4e73bb9cca620676f6cf (DEFAULT_TABLE table=test-table [id=19f1f35da8324795a915fc7e6cc51e97]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:02:04.419836 16705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ba7192adfca4e73bb9cca620676f6cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:02:04.422366 16780 tablet_bootstrap.cc:492] T 4ba7192adfca4e73bb9cca620676f6cf P e0cc0328540d4a9db537a475b14a4ab3: Bootstrap starting.
I20260924 05:02:04.422928 16780 tablet_bootstrap.cc:654] T 4ba7192adfca4e73bb9cca620676f6cf P e0cc0328540d4a9db537a475b14a4ab3: Neither blocks nor log segments found. Creating new log.
I20260924 05:02:04.423348 16780 log.cc:824] T 4ba7192adfca4e73bb9cca620676f6cf P e0cc0328540d4a9db537a475b14a4ab3: Log is configured to *not* fsync() on all Append() calls
I20260924 05:02:04.423982 16780 tablet_bootstrap.cc:492] T 4ba7192adfca4e73bb9cca620676f6cf P e0cc0328540d4a9db537a475b14a4ab3: No bootstrap required, opened a new log
I20260924 05:02:04.424033 16780 ts_tablet_manager.cc:1397] T 4ba7192adfca4e73bb9cca620676f6cf P e0cc0328540d4a9db537a475b14a4ab3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 05:02:04.425267 16780 raft_consensus.cc:348] T 4ba7192adfca4e73bb9cca620676f6cf P e0cc0328540d4a9db537a475b14a4ab3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cc0328540d4a9db537a475b14a4ab3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45917 } } peers { permanent_uuid: "6590438bffac4f5d8023294a6d2fd4ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33823 } } peers { permanent_uuid: "1d219c86d9ea4b6aa6d894eb20867698" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42831 } }
I20260924 05:02:04.425410 16780 raft_consensus.cc:374] T 4ba7192adfca4e73bb9cca620676f6cf P e0cc0328540d4a9db537a475b14a4ab3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:02:04.425487 16780 raft_consensus.cc:729] T 4ba7192adfca4e73bb9cca620676f6cf P e0cc0328540d4a9db537a475b14a4ab3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0cc0328540d4a9db537a475b14a4ab3, State: Initialized, Role: FOLLOWER
I20260924 05:02:04.425606 16780 consensus_queue.cc:260] T 4ba7192adfca4e73bb9cca620676f6cf P e0cc0328540d4a9db537a475b14a4ab3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cc0328540d4a9db537a475b14a4ab3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45917 } } peers { permanent_uuid: "6590438bffac4f5d8023294a6d2fd4ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33823 } } peers { permanent_uuid: "1d219c86d9ea4b6aa6d894eb20867698" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42831 } }
I20260924 05:02:04.425841 16780 ts_tablet_manager.cc:1428] T 4ba7192adfca4e73bb9cca620676f6cf P e0cc0328540d4a9db537a475b14a4ab3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 05:02:04.425855 16771 heartbeater.cc:503] Master 127.0.98.62:41625 was elected leader, sending a full tablet report...
I20260924 05:02:04.484269 16638 raft_consensus.cc:482] T 4ba7192adfca4e73bb9cca620676f6cf P 1d219c86d9ea4b6aa6d894eb20867698 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 05:02:04.484411 16638 raft_consensus.cc:504] T 4ba7192adfca4e73bb9cca620676f6cf P 1d219c86d9ea4b6aa6d894eb20867698 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cc0328540d4a9db537a475b14a4ab3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45917 } } peers { permanent_uuid: "6590438bffac4f5d8023294a6d2fd4ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33823 } } peers { permanent_uuid: "1d219c86d9ea4b6aa6d894eb20867698" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42831 } }
I20260924 05:02:04.484735 16638 leader_election.cc:288] T 4ba7192adfca4e73bb9cca620676f6cf P 1d219c86d9ea4b6aa6d894eb20867698 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0cc0328540d4a9db537a475b14a4ab3 (127.0.98.3:45917), 6590438bffac4f5d8023294a6d2fd4ff (127.0.98.1:33823)
I20260924 05:02:04.488431 16777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:04.485039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55311 (local address 127.0.98.3:45917)
0924 05:02:04.485084 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 05:02:04.485086 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:02:04.485094 (+ 8us) server_negotiation.cc:407] Connection header received
0924 05:02:04.485118 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:02:04.485119 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:02:04.485129 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:02:04.485180 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:02:04.485331 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:04.486102 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:04.487489 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:04.488072 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:04.488141 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:04.488151 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 05:02:04.488323 (+ 172us) server_negotiation.cc:299] Negotiation successful
0924 05:02:04.488359 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 05:02:04.488447 16504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:04.484949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50561 (local address 127.0.98.1:33823)
0924 05:02:04.484993 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 05:02:04.484995 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:02:04.485251 (+ 256us) server_negotiation.cc:407] Connection header received
0924 05:02:04.485292 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:02:04.485293 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:02:04.485318 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:02:04.485384 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:02:04.485519 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:04.486213 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:04.487487 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:04.488087 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:04.488139 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:04.488211 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 05:02:04.488323 (+ 112us) server_negotiation.cc:299] Negotiation successful
0924 05:02:04.488363 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 05:02:04.488463 16639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:04.484908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45917 (local address 127.0.98.2:55311)
0924 05:02:04.484946 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:04.484957 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 05:02:04.485007 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:02:04.485197 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:02:04.485198 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:02:04.485199 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:02:04.485272 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:04.485274 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:04.486183 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:04.486184 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:04.487409 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:04.487412 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:04.488146 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:04.488147 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:04.488233 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:04.488259 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 05:02:04.488313 (+ 54us) client_negotiation.cc:240] Negotiation successful
0924 05:02:04.488361 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20,"mutex_wait_us":32}
I20260924 05:02:04.488476 16783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:04.484908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33823 (local address 127.0.98.2:50561)
0924 05:02:04.485185 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:04.485194 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:02:04.485246 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:02:04.485386 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:02:04.485386 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:02:04.485387 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:02:04.485459 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:04.485461 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:04.486275 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:04.486276 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:04.487414 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:04.487416 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:04.488163 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:04.488164 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:04.488234 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:04.488259 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 05:02:04.488313 (+ 54us) client_negotiation.cc:240] Negotiation successful
0924 05:02:04.488361 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"mutex_wait_us":149,"thread_start_us":69,"threads_started":1}
I20260924 05:02:04.488813 16725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ba7192adfca4e73bb9cca620676f6cf" candidate_uuid: "1d219c86d9ea4b6aa6d894eb20867698" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0cc0328540d4a9db537a475b14a4ab3" is_pre_election: true
I20260924 05:02:04.488971 16725 raft_consensus.cc:2454] T 4ba7192adfca4e73bb9cca620676f6cf P e0cc0328540d4a9db537a475b14a4ab3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d219c86d9ea4b6aa6d894eb20867698 in term 0.
I20260924 05:02:04.489171 16524 leader_election.cc:302] T 4ba7192adfca4e73bb9cca620676f6cf P 1d219c86d9ea4b6aa6d894eb20867698 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d219c86d9ea4b6aa6d894eb20867698, e0cc0328540d4a9db537a475b14a4ab3; no voters:
I20260924 05:02:04.489138 16450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ba7192adfca4e73bb9cca620676f6cf" candidate_uuid: "1d219c86d9ea4b6aa6d894eb20867698" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6590438bffac4f5d8023294a6d2fd4ff" is_pre_election: true
I20260924 05:02:04.489281 16450 raft_consensus.cc:2454] T 4ba7192adfca4e73bb9cca620676f6cf P 6590438bffac4f5d8023294a6d2fd4ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d219c86d9ea4b6aa6d894eb20867698 in term 0.
I20260924 05:02:04.489290 16638 raft_consensus.cc:2781] T 4ba7192adfca4e73bb9cca620676f6cf P 1d219c86d9ea4b6aa6d894eb20867698 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 05:02:04.489362 16638 raft_consensus.cc:482] T 4ba7192adfca4e73bb9cca620676f6cf P 1d219c86d9ea4b6aa6d894eb20867698 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 05:02:04.489393 16638 raft_consensus.cc:3037] T 4ba7192adfca4e73bb9cca620676f6cf P 1d219c86d9ea4b6aa6d894eb20867698 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:02:04.490079 16638 raft_consensus.cc:504] T 4ba7192adfca4e73bb9cca620676f6cf P 1d219c86d9ea4b6aa6d894eb20867698 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cc0328540d4a9db537a475b14a4ab3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45917 } } peers { permanent_uuid: "6590438bffac4f5d8023294a6d2fd4ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33823 } } peers { permanent_uuid: "1d219c86d9ea4b6aa6d894eb20867698" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42831 } }
I20260924 05:02:04.490190 16638 leader_election.cc:288] T 4ba7192adfca4e73bb9cca620676f6cf P 1d219c86d9ea4b6aa6d894eb20867698 [CANDIDATE]: Term 1 election: Requested vote from peers e0cc0328540d4a9db537a475b14a4ab3 (127.0.98.3:45917), 6590438bffac4f5d8023294a6d2fd4ff (127.0.98.1:33823)
I20260924 05:02:04.490350 16450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ba7192adfca4e73bb9cca620676f6cf" candidate_uuid: "1d219c86d9ea4b6aa6d894eb20867698" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6590438bffac4f5d8023294a6d2fd4ff"
I20260924 05:02:04.490351 16725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ba7192adfca4e73bb9cca620676f6cf" candidate_uuid: "1d219c86d9ea4b6aa6d894eb20867698" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0cc0328540d4a9db537a475b14a4ab3"
I20260924 05:02:04.490419 16450 raft_consensus.cc:3037] T 4ba7192adfca4e73bb9cca620676f6cf P 6590438bffac4f5d8023294a6d2fd4ff [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:02:04.490422 16725 raft_consensus.cc:3037] T 4ba7192adfca4e73bb9cca620676f6cf P e0cc0328540d4a9db537a475b14a4ab3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:02:04.491048 16450 raft_consensus.cc:2454] T 4ba7192adfca4e73bb9cca620676f6cf P 6590438bffac4f5d8023294a6d2fd4ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d219c86d9ea4b6aa6d894eb20867698 in term 1.
I20260924 05:02:04.491110 16725 raft_consensus.cc:2454] T 4ba7192adfca4e73bb9cca620676f6cf P e0cc0328540d4a9db537a475b14a4ab3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d219c86d9ea4b6aa6d894eb20867698 in term 1.
I20260924 05:02:04.491227 16523 leader_election.cc:302] T 4ba7192adfca4e73bb9cca620676f6cf P 1d219c86d9ea4b6aa6d894eb20867698 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d219c86d9ea4b6aa6d894eb20867698, 6590438bffac4f5d8023294a6d2fd4ff; no voters:
I20260924 05:02:04.491322 16638 raft_consensus.cc:2781] T 4ba7192adfca4e73bb9cca620676f6cf P 1d219c86d9ea4b6aa6d894eb20867698 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:02:04.491474 16638 raft_consensus.cc:686] T 4ba7192adfca4e73bb9cca620676f6cf P 1d219c86d9ea4b6aa6d894eb20867698 [term 1 LEADER]: Becoming Leader. State: Replica: 1d219c86d9ea4b6aa6d894eb20867698, State: Running, Role: LEADER
I20260924 05:02:04.491533 16638 consensus_queue.cc:237] T 4ba7192adfca4e73bb9cca620676f6cf P 1d219c86d9ea4b6aa6d894eb20867698 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cc0328540d4a9db537a475b14a4ab3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45917 } } peers { permanent_uuid: "6590438bffac4f5d8023294a6d2fd4ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33823 } } peers { permanent_uuid: "1d219c86d9ea4b6aa6d894eb20867698" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42831 } }
I20260924 05:02:04.492300 16308 catalog_manager.cc:5495] T 4ba7192adfca4e73bb9cca620676f6cf P 1d219c86d9ea4b6aa6d894eb20867698 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d219c86d9ea4b6aa6d894eb20867698 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "1d219c86d9ea4b6aa6d894eb20867698" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cc0328540d4a9db537a475b14a4ab3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6590438bffac4f5d8023294a6d2fd4ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33823 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d219c86d9ea4b6aa6d894eb20867698" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42831 } health_report { overall_health: HEALTHY } } }
W20260924 05:02:04.499733 16637 tablet.cc:2369] T 4ba7192adfca4e73bb9cca620676f6cf P 1d219c86d9ea4b6aa6d894eb20867698: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 05:02:04.510699 16640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:04.508376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56292 (local address 127.0.98.2:42831)
0924 05:02:04.508438 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 05:02:04.508440 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:02:04.508449 (+ 9us) server_negotiation.cc:407] Connection header received
0924 05:02:04.508489 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:02:04.508490 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:02:04.508510 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:02:04.508572 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:02:04.508675 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:04.509163 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:04.509924 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:04.510186 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:04.510237 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:04.510388 (+ 151us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:02:04.510505 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:02:04.510567 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 05:02:04.510601 (+ 34us) server_negotiation.cc:299] Negotiation successful
0924 05:02:04.510621 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 05:02:04.512097 16450 raft_consensus.cc:1261] T 4ba7192adfca4e73bb9cca620676f6cf P 6590438bffac4f5d8023294a6d2fd4ff [term 1 FOLLOWER]: Refusing update from remote peer 1d219c86d9ea4b6aa6d894eb20867698: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:02:04.512099 16725 raft_consensus.cc:1261] T 4ba7192adfca4e73bb9cca620676f6cf P e0cc0328540d4a9db537a475b14a4ab3 [term 1 FOLLOWER]: Refusing update from remote peer 1d219c86d9ea4b6aa6d894eb20867698: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:02:04.512382 16784 consensus_queue.cc:1035] T 4ba7192adfca4e73bb9cca620676f6cf P 1d219c86d9ea4b6aa6d894eb20867698 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6590438bffac4f5d8023294a6d2fd4ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:02:04.512509 16638 consensus_queue.cc:1035] T 4ba7192adfca4e73bb9cca620676f6cf P 1d219c86d9ea4b6aa6d894eb20867698 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0cc0328540d4a9db537a475b14a4ab3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:02:04.514492 16787 mvcc.cc:205] Tried to move back new op lower bound from 7332766205999136768 to 7332766205917958144. Current Snapshot: MvccSnapshot[applied={T|T < 7332766205999136768}]
I20260924 05:02:04.514719 16789 mvcc.cc:205] Tried to move back new op lower bound from 7332766205999136768 to 7332766205917958144. Current Snapshot: MvccSnapshot[applied={T|T < 7332766205999136768}]
I20260924 05:02:04.516921 16306 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35434:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:02:04.517004 16306 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35434:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:02:04.517678 16306 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e45cc94f6d2f434f9e8cf8dd78e8dff7: Sending DeleteTablet for 3 replicas of tablet 4ba7192adfca4e73bb9cca620676f6cf
I20260924 05:02:04.517958 16430 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ba7192adfca4e73bb9cca620676f6cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:02:04 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38212
I20260924 05:02:04.517951 16705 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ba7192adfca4e73bb9cca620676f6cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:02:04 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55698
I20260924 05:02:04.518057 16505 tablet_replica.cc:330] T 4ba7192adfca4e73bb9cca620676f6cf P 6590438bffac4f5d8023294a6d2fd4ff: stopping tablet replica
I20260924 05:02:04.518108 16505 raft_consensus.cc:2229] T 4ba7192adfca4e73bb9cca620676f6cf P 6590438bffac4f5d8023294a6d2fd4ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:02:04.518116 16570 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ba7192adfca4e73bb9cca620676f6cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:02:04 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56286
I20260924 05:02:04.518158 16505 raft_consensus.cc:2258] T 4ba7192adfca4e73bb9cca620676f6cf P 6590438bffac4f5d8023294a6d2fd4ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:02:04.518193 16642 tablet_replica.cc:330] T 4ba7192adfca4e73bb9cca620676f6cf P 1d219c86d9ea4b6aa6d894eb20867698: stopping tablet replica
I20260924 05:02:04.518186 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 16346
I20260924 05:02:04.518239 16642 raft_consensus.cc:2229] T 4ba7192adfca4e73bb9cca620676f6cf P 1d219c86d9ea4b6aa6d894eb20867698 [term 1 LEADER]: Raft consensus shutting down.
I20260924 05:02:04.518354 16642 raft_consensus.cc:2258] T 4ba7192adfca4e73bb9cca620676f6cf P 1d219c86d9ea4b6aa6d894eb20867698 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:02:04.518795 16642 ts_tablet_manager.cc:1905] T 4ba7192adfca4e73bb9cca620676f6cf P 1d219c86d9ea4b6aa6d894eb20867698: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:02:04.520491 16642 ts_tablet_manager.cc:1918] T 4ba7192adfca4e73bb9cca620676f6cf P 1d219c86d9ea4b6aa6d894eb20867698: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 05:02:04.520550 16642 log.cc:1196] T 4ba7192adfca4e73bb9cca620676f6cf P 1d219c86d9ea4b6aa6d894eb20867698: Deleting WAL directory at /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-1/wal/wals/4ba7192adfca4e73bb9cca620676f6cf
I20260924 05:02:04.520736 16642 ts_tablet_manager.cc:1939] T 4ba7192adfca4e73bb9cca620676f6cf P 1d219c86d9ea4b6aa6d894eb20867698: Deleting consensus metadata
I20260924 05:02:04.521026 16294 catalog_manager.cc:4841] TS 1d219c86d9ea4b6aa6d894eb20867698 (127.0.98.2:42831): tablet 4ba7192adfca4e73bb9cca620676f6cf (table test-table [id=19f1f35da8324795a915fc7e6cc51e97]) successfully deleted
W20260924 05:02:04.523769 16294 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33823 (error 108)
I20260924 05:02:04.524032 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 16502
I20260924 05:02:04.525264 16797 tablet_replica.cc:330] T 4ba7192adfca4e73bb9cca620676f6cf P e0cc0328540d4a9db537a475b14a4ab3: stopping tablet replica
I20260924 05:02:04.525391 16797 raft_consensus.cc:2229] T 4ba7192adfca4e73bb9cca620676f6cf P e0cc0328540d4a9db537a475b14a4ab3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:02:04.525547 16797 raft_consensus.cc:2258] T 4ba7192adfca4e73bb9cca620676f6cf P e0cc0328540d4a9db537a475b14a4ab3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:02:04.525638 16503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:04.525510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33823 (local address 127.0.0.1:38218)
0924 05:02:04.525553 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:04.525602 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260924 05:02:04.525961 16797 ts_tablet_manager.cc:1905] T 4ba7192adfca4e73bb9cca620676f6cf P e0cc0328540d4a9db537a475b14a4ab3: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 05:02:04.525817 16294 catalog_manager.cc:4584] TS 6590438bffac4f5d8023294a6d2fd4ff (127.0.98.1:33823): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4ba7192adfca4e73bb9cca620676f6cf: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33823: connect: Connection refused (error 111)
I20260924 05:02:04.527597 16797 ts_tablet_manager.cc:1918] T 4ba7192adfca4e73bb9cca620676f6cf P e0cc0328540d4a9db537a475b14a4ab3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 05:02:04.527655 16797 log.cc:1196] T 4ba7192adfca4e73bb9cca620676f6cf P e0cc0328540d4a9db537a475b14a4ab3: Deleting WAL directory at /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-2/wal/wals/4ba7192adfca4e73bb9cca620676f6cf
I20260924 05:02:04.527878 16797 ts_tablet_manager.cc:1939] T 4ba7192adfca4e73bb9cca620676f6cf P e0cc0328540d4a9db537a475b14a4ab3: Deleting consensus metadata
I20260924 05:02:04.528286 16295 catalog_manager.cc:4841] TS e0cc0328540d4a9db537a475b14a4ab3 (127.0.98.3:45917): tablet 4ba7192adfca4e73bb9cca620676f6cf (table test-table [id=19f1f35da8324795a915fc7e6cc51e97]) successfully deleted
I20260924 05:02:04.528589 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 16641
I20260924 05:02:04.532348 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 16275
I20260924 05:02:24.536882 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41625
--webserver_interface=127.0.98.62
--webserver_port=43227
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41625
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:02:24.615098 16799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:02:24.615309 16799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:02:24.615348 16799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:02:24.616496 16799 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 05:02:24.616534 16799 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 05:02:24.616549 16799 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 05:02:24.616564 16799 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:02:24.616580 16799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:02:24.616598 16799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:02:24.616617 16799 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:02:24.616641 16799 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:02:24.618129 16799 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41625
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=43227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16799
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:02:24.618374 16799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:02:24.618582 16799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:02:24.623230 16804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:02:24.623315 16807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:02:24.623481 16799 server_base.cc:1034] running on GCE node
W20260924 05:02:24.623236 16805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:02:24.623720 16799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:02:24.624019 16799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:02:24.625190 16799 hybrid_clock.cc:648] HybridClock initialized: now 1790226144625169 us; error 33 us; skew 500 ppm
Sep 24 05:02:24 dist-test-slave-70st krb5kdc[15731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226144, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:02:24.627630 16799 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 05:02:24.628047 16799 webserver.cc:460] Webserver started at http://127.0.98.62:43227/ using document root <none> and password file <none>
I20260924 05:02:24.628218 16799 fs_manager.cc:359] Metadata directory not provided
I20260924 05:02:24.628264 16799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:02:24.629544 16799 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 05:02:24.630168 16814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:02:24.630347 16799 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:02:24.630439 16799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/master-0/wal
uuid: "e45cc94f6d2f434f9e8cf8dd78e8dff7"
format_stamp: "Formatted at 2026-09-24 05:01:43 on dist-test-slave-70st"
server_key: "bc285d33076a5410016734cc94a222b3"
server_key_iv: "9a02138c971c7442b414f0d78a1bcd86"
server_key_version: "encryptionkey@0"
I20260924 05:02:24.630779 16799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:02:24.649289 16799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:02:24.651036 16799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:02:24.651155 16799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:02:24.655007 16799 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41625
I20260924 05:02:24.655043 16866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41625 every 8 connection(s)
I20260924 05:02:24.655442 16799 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:02:24.656791 16867 sys_catalog.cc:263] Verifying existing consensus state
I20260924 05:02:24.657366 16867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e45cc94f6d2f434f9e8cf8dd78e8dff7: Bootstrap starting.
I20260924 05:02:24.661661 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 16799
I20260924 05:02:24.662030 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:33823
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:41625
--webserver_port=44521
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 05:02:24.665781 16867 log.cc:824] T 00000000000000000000000000000000 P e45cc94f6d2f434f9e8cf8dd78e8dff7: Log is configured to *not* fsync() on all Append() calls
I20260924 05:02:24.667841 16867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e45cc94f6d2f434f9e8cf8dd78e8dff7: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 05:02:24.668109 16867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e45cc94f6d2f434f9e8cf8dd78e8dff7: Bootstrap complete.
I20260924 05:02:24.670172 16867 raft_consensus.cc:348] T 00000000000000000000000000000000 P e45cc94f6d2f434f9e8cf8dd78e8dff7 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e45cc94f6d2f434f9e8cf8dd78e8dff7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41625 } }
I20260924 05:02:24.670392 16867 raft_consensus.cc:729] T 00000000000000000000000000000000 P e45cc94f6d2f434f9e8cf8dd78e8dff7 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: e45cc94f6d2f434f9e8cf8dd78e8dff7, State: Initialized, Role: FOLLOWER
I20260924 05:02:24.670558 16867 consensus_queue.cc:260] T 00000000000000000000000000000000 P e45cc94f6d2f434f9e8cf8dd78e8dff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e45cc94f6d2f434f9e8cf8dd78e8dff7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41625 } }
I20260924 05:02:24.670634 16867 raft_consensus.cc:388] T 00000000000000000000000000000000 P e45cc94f6d2f434f9e8cf8dd78e8dff7 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:02:24.670681 16867 raft_consensus.cc:482] T 00000000000000000000000000000000 P e45cc94f6d2f434f9e8cf8dd78e8dff7 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:02:24.670725 16867 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e45cc94f6d2f434f9e8cf8dd78e8dff7 [term 2 FOLLOWER]: Advancing to term 3
I20260924 05:02:24.671661 16867 raft_consensus.cc:504] T 00000000000000000000000000000000 P e45cc94f6d2f434f9e8cf8dd78e8dff7 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e45cc94f6d2f434f9e8cf8dd78e8dff7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41625 } }
I20260924 05:02:24.671752 16867 leader_election.cc:302] T 00000000000000000000000000000000 P e45cc94f6d2f434f9e8cf8dd78e8dff7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e45cc94f6d2f434f9e8cf8dd78e8dff7; no voters:
I20260924 05:02:24.671905 16867 leader_election.cc:288] T 00000000000000000000000000000000 P e45cc94f6d2f434f9e8cf8dd78e8dff7 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 05:02:24.671975 16871 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e45cc94f6d2f434f9e8cf8dd78e8dff7 [term 3 FOLLOWER]: Leader election won for term 3
I20260924 05:02:24.672152 16867 sys_catalog.cc:564] T 00000000000000000000000000000000 P e45cc94f6d2f434f9e8cf8dd78e8dff7 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:02:24.672189 16871 raft_consensus.cc:686] T 00000000000000000000000000000000 P e45cc94f6d2f434f9e8cf8dd78e8dff7 [term 3 LEADER]: Becoming Leader. State: Replica: e45cc94f6d2f434f9e8cf8dd78e8dff7, State: Running, Role: LEADER
I20260924 05:02:24.672291 16871 consensus_queue.cc:237] T 00000000000000000000000000000000 P e45cc94f6d2f434f9e8cf8dd78e8dff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e45cc94f6d2f434f9e8cf8dd78e8dff7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41625 } }
I20260924 05:02:24.672658 16871 sys_catalog.cc:455] T 00000000000000000000000000000000 P e45cc94f6d2f434f9e8cf8dd78e8dff7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e45cc94f6d2f434f9e8cf8dd78e8dff7. Latest consensus state: current_term: 3 leader_uuid: "e45cc94f6d2f434f9e8cf8dd78e8dff7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e45cc94f6d2f434f9e8cf8dd78e8dff7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41625 } } }
I20260924 05:02:24.672739 16871 sys_catalog.cc:458] T 00000000000000000000000000000000 P e45cc94f6d2f434f9e8cf8dd78e8dff7 [sys.catalog]: This master's current role is: LEADER
I20260924 05:02:24.672943 16872 sys_catalog.cc:455] T 00000000000000000000000000000000 P e45cc94f6d2f434f9e8cf8dd78e8dff7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "e45cc94f6d2f434f9e8cf8dd78e8dff7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e45cc94f6d2f434f9e8cf8dd78e8dff7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41625 } } }
I20260924 05:02:24.673009 16872 sys_catalog.cc:458] T 00000000000000000000000000000000 P e45cc94f6d2f434f9e8cf8dd78e8dff7 [sys.catalog]: This master's current role is: LEADER
I20260924 05:02:24.674203 16887 catalog_manager.cc:1259] Loaded cluster ID: 31fbd21556764b1e891939dae73550c7
I20260924 05:02:24.674248 16887 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e45cc94f6d2f434f9e8cf8dd78e8dff7: loading cluster ID for follower catalog manager: success
I20260924 05:02:24.675379 16887 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e45cc94f6d2f434f9e8cf8dd78e8dff7: acquiring CA information for follower catalog manager: success
F20260924 05:02:24.675745 16887 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7efedc6919ad google::LogMessage::Fail() at ??:0
@ 0x7efedc69488a google::LogMessage::SendToLog() at ??:0
@ 0x7efedc6914de google::LogMessage::Flush() at ??:0
@ 0x7efedc692a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7efededfe104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7efee45ea6b9 kudu::master::CatalogManager::LoadTspkEntries() at ??:0
@ 0x7efee45ea989 kudu::master::CatalogManager::PrepareFollowerTokenVerifier() at ??:0
@ 0x7efee45eae4e kudu::master::CatalogManager::PrepareFollower() at ??:0
@ 0x7efee45ed8c2 kudu::master::CatalogManagerBgTasks::Run() at ??:0
@ 0x7efede554954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7efedf3b86db start_thread at ??:0
@ 0x7efedb18f61f clone at ??:0
W20260924 05:02:24.756363 16869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:02:24.756564 16869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:02:24.756587 16869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:02:24.757742 16869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:02:24.757779 16869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:02:24.757804 16869 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 05:02:24.759218 16869 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:33823
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=44521
--tserver_master_addrs=127.0.98.62:41625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16869
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:02:24.759441 16869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:02:24.759658 16869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:02:24.765503 16905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:02:24.765503 16906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:02:24.765524 16908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:02:24.765795 16869 server_base.cc:1034] running on GCE node
I20260924 05:02:24.766047 16869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:02:24.766294 16869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:02:24.767452 16869 hybrid_clock.cc:648] HybridClock initialized: now 1790226144767429 us; error 39 us; skew 500 ppm
Sep 24 05:02:24 dist-test-slave-70st krb5kdc[15731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226144, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:02:24.776101 16869 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 05:02:24.776566 16869 webserver.cc:460] Webserver started at http://127.0.98.1:44521/ using document root <none> and password file <none>
I20260924 05:02:24.776798 16869 fs_manager.cc:359] Metadata directory not provided
I20260924 05:02:24.776846 16869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:02:24.778319 16869 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260924 05:02:24.779091 16915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:02:24.779299 16869 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 05:02:24.779412 16869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-0/wal
uuid: "6590438bffac4f5d8023294a6d2fd4ff"
format_stamp: "Formatted at 2026-09-24 05:01:43 on dist-test-slave-70st"
server_key: "b4f01fdb4ff7c2a227ab16508d224509"
server_key_iv: "359943cd0bcef6f5b6d2416d8cd8b768"
server_key_version: "encryptionkey@0"
I20260924 05:02:24.779759 16869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:02:24.788379 16869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:02:24.790521 16869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:02:24.790650 16869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:02:24.790863 16869 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:02:24.791316 16922 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 05:02:24.792846 16869 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 05:02:24.792900 16869 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260924 05:02:24.792924 16869 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 05:02:24.793605 16869 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 05:02:24.793648 16869 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.001s sys 0.000s
I20260924 05:02:24.793700 16922 tablet_bootstrap.cc:492] T 4ba7192adfca4e73bb9cca620676f6cf P 6590438bffac4f5d8023294a6d2fd4ff: Bootstrap starting.
I20260924 05:02:24.800806 16869 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33823
I20260924 05:02:24.801308 16869 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-0/data/info.pb
I20260924 05:02:24.801862 17029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33823 every 8 connection(s)
I20260924 05:02:24.803858 16922 log.cc:824] T 4ba7192adfca4e73bb9cca620676f6cf P 6590438bffac4f5d8023294a6d2fd4ff: Log is configured to *not* fsync() on all Append() calls
I20260924 05:02:24.805929 16922 tablet_bootstrap.cc:492] T 4ba7192adfca4e73bb9cca620676f6cf P 6590438bffac4f5d8023294a6d2fd4ff: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 05:02:24.806164 16922 tablet_bootstrap.cc:492] T 4ba7192adfca4e73bb9cca620676f6cf P 6590438bffac4f5d8023294a6d2fd4ff: Bootstrap complete.
I20260924 05:02:24.806864 16922 ts_tablet_manager.cc:1397] T 4ba7192adfca4e73bb9cca620676f6cf P 6590438bffac4f5d8023294a6d2fd4ff: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.006s
I20260924 05:02:24.807134 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 16869
I20260924 05:02:24.807444 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:42831
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:41625
--webserver_port=44059
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 05:02:24.807942 16922 raft_consensus.cc:348] T 4ba7192adfca4e73bb9cca620676f6cf P 6590438bffac4f5d8023294a6d2fd4ff [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cc0328540d4a9db537a475b14a4ab3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45917 } } peers { permanent_uuid: "6590438bffac4f5d8023294a6d2fd4ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33823 } } peers { permanent_uuid: "1d219c86d9ea4b6aa6d894eb20867698" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42831 } }
I20260924 05:02:24.808172 16922 raft_consensus.cc:729] T 4ba7192adfca4e73bb9cca620676f6cf P 6590438bffac4f5d8023294a6d2fd4ff [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6590438bffac4f5d8023294a6d2fd4ff, State: Initialized, Role: FOLLOWER
I20260924 05:02:24.808302 16922 consensus_queue.cc:260] T 4ba7192adfca4e73bb9cca620676f6cf P 6590438bffac4f5d8023294a6d2fd4ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cc0328540d4a9db537a475b14a4ab3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45917 } } peers { permanent_uuid: "6590438bffac4f5d8023294a6d2fd4ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33823 } } peers { permanent_uuid: "1d219c86d9ea4b6aa6d894eb20867698" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42831 } }
I20260924 05:02:24.808547 16922 ts_tablet_manager.cc:1428] T 4ba7192adfca4e73bb9cca620676f6cf P 6590438bffac4f5d8023294a6d2fd4ff: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
W20260924 05:02:24.854794 17032 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:02:24.801982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41625 (local address 127.0.98.1:35765)
0924 05:02:24.802151 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:24.802194 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 05:02:24.802654 (+ 460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:02:24.854621 (+ 51967us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41625: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":93,"threads_started":1}
W20260924 05:02:24.855068 17030 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41625 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:41625: Client connection negotiation failed: client connection to 127.0.98.62:41625: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 05:02:24.855410 17032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:24.855278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41625 (local address 127.0.98.1:54063)
0924 05:02:24.855324 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:24.855387 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260924 05:02:24.855768 17032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:24.855692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41625 (local address 127.0.98.1:37875)
0924 05:02:24.855730 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:24.855753 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260924 05:02:24.855861 17030 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 05:02:24.886973 17035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:02:24.887166 17035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:02:24.887188 17035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:02:24.888248 17035 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:02:24.888283 17035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:02:24.888306 17035 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 05:02:24.889703 17035 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:42831
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=44059
--tserver_master_addrs=127.0.98.62:41625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17035
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:02:24.889890 17035 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:02:24.890067 17035 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:02:24.894546 17041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:02:24.894595 17040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:02:24.894564 17043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:02:24.894749 17035 server_base.cc:1034] running on GCE node
I20260924 05:02:24.894986 17035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:02:24.895205 17035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:02:24.896354 17035 hybrid_clock.cc:648] HybridClock initialized: now 1790226144896338 us; error 29 us; skew 500 ppm
Sep 24 05:02:24 dist-test-slave-70st krb5kdc[15731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226144, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:02:24.898679 17035 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 05:02:24.899091 17035 webserver.cc:460] Webserver started at http://127.0.98.2:44059/ using document root <none> and password file <none>
I20260924 05:02:24.899288 17035 fs_manager.cc:359] Metadata directory not provided
I20260924 05:02:24.899336 17035 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:02:24.900553 17035 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 05:02:24.901026 17050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:02:24.901170 17035 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 05:02:24.901243 17035 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-1/wal
uuid: "1d219c86d9ea4b6aa6d894eb20867698"
format_stamp: "Formatted at 2026-09-24 05:01:43 on dist-test-slave-70st"
server_key: "9182aeba844e056977f6da6dfdfdbd18"
server_key_iv: "0cc71887ec5726087abc5e4c97c68e26"
server_key_version: "encryptionkey@0"
I20260924 05:02:24.901623 17035 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:02:24.915830 17035 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:02:24.917919 17035 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:02:24.918054 17035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:02:24.918259 17035 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:02:24.918538 17035 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:02:24.918572 17035 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:02:24.918601 17035 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:02:24.918622 17035 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:02:24.924464 17035 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42831
I20260924 05:02:24.924504 17163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42831 every 8 connection(s)
I20260924 05:02:24.924819 17035 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-1/data/info.pb
I20260924 05:02:24.925864 17166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:24.925519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41625 (local address 127.0.98.2:45457)
0924 05:02:24.925655 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:24.925721 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":67,"threads_started":1}
W20260924 05:02:24.926222 17164 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41625 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:41625: Client connection negotiation failed: client connection to 127.0.98.62:41625: connect: Connection refused (error 111)
I20260924 05:02:24.926474 17166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:24.926395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41625 (local address 127.0.98.2:33455)
0924 05:02:24.926432 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:24.926456 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260924 05:02:24.926666 17166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:24.926611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41625 (local address 127.0.98.2:53681)
0924 05:02:24.926638 (+ 27us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:24.926654 (+ 16us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260924 05:02:24.926745 17164 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 05:02:24.932382 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 17035
I20260924 05:02:24.932655 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:45917
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:41625
--webserver_port=40245
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:02:25.012384 17167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:02:25.012595 17167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:02:25.012619 17167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:02:25.013846 17167 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:02:25.013891 17167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:02:25.013917 17167 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 05:02:25.015415 17167 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:45917
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=40245
--tserver_master_addrs=127.0.98.62:41625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17167
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:02:25.015651 17167 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:02:25.015874 17167 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:02:25.020759 17173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:02:25.020744 17172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:02:25.020987 17167 server_base.cc:1034] running on GCE node
W20260924 05:02:25.020747 17175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:02:25.021209 17167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:02:25.021477 17167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:02:25.022634 17167 hybrid_clock.cc:648] HybridClock initialized: now 1790226145022613 us; error 34 us; skew 500 ppm
Sep 24 05:02:25 dist-test-slave-70st krb5kdc[15731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226145, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:02:25.025740 17167 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 05:02:25.026309 17167 webserver.cc:460] Webserver started at http://127.0.98.3:40245/ using document root <none> and password file <none>
I20260924 05:02:25.026544 17167 fs_manager.cc:359] Metadata directory not provided
I20260924 05:02:25.026587 17167 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:02:25.028154 17167 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 05:02:25.028697 17182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:02:25.028834 17167 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 05:02:25.028896 17167 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-2/wal
uuid: "e0cc0328540d4a9db537a475b14a4ab3"
format_stamp: "Formatted at 2026-09-24 05:01:43 on dist-test-slave-70st"
server_key: "8e0ce449eaf3849a57093216055d4807"
server_key_iv: "c4d89a7ff885b5cdf2cb5321864d74dc"
server_key_version: "encryptionkey@0"
I20260924 05:02:25.029239 17167 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:02:25.042320 17167 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:02:25.044257 17167 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:02:25.044369 17167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:02:25.044577 17167 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:02:25.044868 17167 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:02:25.044901 17167 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:02:25.044931 17167 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:02:25.044948 17167 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:02:25.050299 17167 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45917
I20260924 05:02:25.050343 17295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45917 every 8 connection(s)
I20260924 05:02:25.050668 17167 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790225929848825-392-0/minicluster-data/ts-2/data/info.pb
I20260924 05:02:25.051723 17298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:25.051308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41625 (local address 127.0.98.3:49453)
0924 05:02:25.051499 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:25.051575 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":78,"threads_started":1}
W20260924 05:02:25.052186 17296 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41625 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:41625: Client connection negotiation failed: client connection to 127.0.98.62:41625: connect: Connection refused (error 111)
I20260924 05:02:25.052491 17298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:25.052400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41625 (local address 127.0.98.3:57175)
0924 05:02:25.052444 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:25.052471 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260924 05:02:25.052738 17298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:25.052678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41625 (local address 127.0.98.3:37191)
0924 05:02:25.052709 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:25.052726 (+ 17us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260924 05:02:25.052866 17296 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 05:02:25.057183 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 17167
I20260924 05:02:25.856487 17300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:25.856210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41625 (local address 127.0.98.1:39235)
0924 05:02:25.856381 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:25.856420 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":67,"threads_started":1}
I20260924 05:02:25.927171 17301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:25.927003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41625 (local address 127.0.98.2:37101)
0924 05:02:25.927096 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:25.927125 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80,"thread_start_us":52,"threads_started":1}
I20260924 05:02:26.046496 17302 raft_consensus.cc:482] T 4ba7192adfca4e73bb9cca620676f6cf P 6590438bffac4f5d8023294a6d2fd4ff [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 05:02:26.046614 17302 raft_consensus.cc:504] T 4ba7192adfca4e73bb9cca620676f6cf P 6590438bffac4f5d8023294a6d2fd4ff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cc0328540d4a9db537a475b14a4ab3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45917 } } peers { permanent_uuid: "6590438bffac4f5d8023294a6d2fd4ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33823 } } peers { permanent_uuid: "1d219c86d9ea4b6aa6d894eb20867698" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42831 } }
I20260924 05:02:26.046883 17302 leader_election.cc:288] T 4ba7192adfca4e73bb9cca620676f6cf P 6590438bffac4f5d8023294a6d2fd4ff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e0cc0328540d4a9db537a475b14a4ab3 (127.0.98.3:45917), 1d219c86d9ea4b6aa6d894eb20867698 (127.0.98.2:42831)
Sep 24 05:02:26 dist-test-slave-70st krb5kdc[15731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226144, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Sep 24 05:02:26 dist-test-slave-70st krb5kdc[15731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226144, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260924 05:02:26.053495 17306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:26.053210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41625 (local address 127.0.98.3:48449)
0924 05:02:26.053360 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:26.053398 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":86,"threads_started":1}
I20260924 05:02:26.057300 17305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:26.047180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40397 (local address 127.0.98.2:42831)
0924 05:02:26.047320 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 05:02:26.047330 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 05:02:26.047358 (+ 28us) server_negotiation.cc:407] Connection header received
0924 05:02:26.047766 (+ 408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:02:26.047768 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:02:26.047812 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:02:26.047932 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:02:26.048640 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:26.049523 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:26.051223 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:26.051762 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:26.051874 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:26.053817 (+ 1943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:02:26.053820 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:02:26.053820 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:02:26.053829 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:02:26.055387 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:02:26.055835 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:02:26.055838 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:02:26.055839 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:02:26.055957 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:02:26.056311 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:02:26.056312 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:02:26.056312 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:02:26.056637 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:02:26.056728 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 05:02:26.057053 (+ 325us) server_negotiation.cc:299] Negotiation successful
0924 05:02:26.057188 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":51,"threads_started":1}
I20260924 05:02:26.057300 17304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:26.047175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56489 (local address 127.0.98.3:45917)
0924 05:02:26.047289 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0924 05:02:26.047302 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 05:02:26.047325 (+ 23us) server_negotiation.cc:407] Connection header received
0924 05:02:26.047742 (+ 417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:02:26.047744 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:02:26.047796 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:02:26.047920 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:02:26.048640 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:26.049536 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:26.050628 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:26.051035 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:26.051137 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:26.053394 (+ 2257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:02:26.053398 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:02:26.053398 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:02:26.053410 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:02:26.055305 (+ 1895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:02:26.055837 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:02:26.055839 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:02:26.055840 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:02:26.055957 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:02:26.056292 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:02:26.056294 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:02:26.056294 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:02:26.056638 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:02:26.056728 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 05:02:26.057053 (+ 325us) server_negotiation.cc:299] Negotiation successful
0924 05:02:26.057188 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78,"thread_start_us":52,"threads_started":1}
I20260924 05:02:26.057310 17303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:26.047090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45917 (local address 127.0.98.1:56489)
0924 05:02:26.047223 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:26.047232 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:02:26.047284 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:02:26.047977 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:02:26.047980 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:02:26.048353 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:02:26.048527 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:26.048533 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:26.049619 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:26.049620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:26.050556 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:26.050559 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:26.051111 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:26.051112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:26.051215 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:26.051710 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:02:26.051720 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:02:26.053280 (+ 1560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:02:26.055381 (+ 2101us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:02:26.055387 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:02:26.055389 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:02:26.055760 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:02:26.056022 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:02:26.056023 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:02:26.056023 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:02:26.056180 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:02:26.056715 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:02:26.056722 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:02:26.056876 (+ 154us) client_negotiation.cc:769] Sending connection context
0924 05:02:26.056996 (+ 120us) client_negotiation.cc:240] Negotiation successful
0924 05:02:26.057179 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"mutex_wait_us":36,"thread_start_us":66,"threads_started":1}
I20260924 05:02:26.057374 17300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:26.047072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42831 (local address 127.0.98.1:40397)
0924 05:02:26.047129 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:26.047138 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:02:26.047190 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:02:26.047978 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:02:26.047980 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:02:26.048344 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:02:26.048527 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:26.048533 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:26.049605 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:26.049606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:26.050553 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:26.050556 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:26.051838 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:26.051839 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:26.051883 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:26.052273 (+ 390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:02:26.052277 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:02:26.053730 (+ 1453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:02:26.055445 (+ 1715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:02:26.055446 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:02:26.055446 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:02:26.055760 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:02:26.056022 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:02:26.056023 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:02:26.056023 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:02:26.056180 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:02:26.056715 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:02:26.056722 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:02:26.056876 (+ 154us) client_negotiation.cc:769] Sending connection context
0924 05:02:26.056996 (+ 120us) client_negotiation.cc:240] Negotiation successful
0924 05:02:26.057244 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":184}
W20260924 05:02:26.058089 16919 leader_election.cc:341] T 4ba7192adfca4e73bb9cca620676f6cf P 6590438bffac4f5d8023294a6d2fd4ff [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e0cc0328540d4a9db537a475b14a4ab3 (127.0.98.3:45917): Not found: Tablet not found: 4ba7192adfca4e73bb9cca620676f6cf
W20260924 05:02:26.058512 16918 leader_election.cc:341] T 4ba7192adfca4e73bb9cca620676f6cf P 6590438bffac4f5d8023294a6d2fd4ff [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1d219c86d9ea4b6aa6d894eb20867698 (127.0.98.2:42831): Not found: Tablet not found: 4ba7192adfca4e73bb9cca620676f6cf
I20260924 05:02:26.058561 16918 leader_election.cc:302] T 4ba7192adfca4e73bb9cca620676f6cf P 6590438bffac4f5d8023294a6d2fd4ff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6590438bffac4f5d8023294a6d2fd4ff; no voters: 1d219c86d9ea4b6aa6d894eb20867698, e0cc0328540d4a9db537a475b14a4ab3
I20260924 05:02:26.058652 17302 raft_consensus.cc:2726] T 4ba7192adfca4e73bb9cca620676f6cf P 6590438bffac4f5d8023294a6d2fd4ff [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 05:02:26.857163 17307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:26.856931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41625 (local address 127.0.98.1:59255)
0924 05:02:26.857080 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:26.857116 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":69,"threads_started":1}
I20260924 05:02:26.927793 17308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:26.927615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41625 (local address 127.0.98.2:34627)
0924 05:02:26.927718 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:26.927754 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":57,"threads_started":1}
I20260924 05:02:27.054136 17309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:27.053926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41625 (local address 127.0.98.3:40311)
0924 05:02:27.054048 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:27.054092 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":59,"threads_started":1}
I20260924 05:02:27.857812 17310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:27.857600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41625 (local address 127.0.98.1:58101)
0924 05:02:27.857727 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:27.857762 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":66,"threads_started":1}
I20260924 05:02:27.928373 17311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:27.928160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41625 (local address 127.0.98.2:49333)
0924 05:02:27.928289 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:27.928323 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":65,"threads_started":1}
I20260924 05:02:28.054757 17312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:28.054530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41625 (local address 127.0.98.3:56103)
0924 05:02:28.054671 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:28.054711 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":58,"threads_started":1}
I20260924 05:02:28.091195 17313 raft_consensus.cc:482] T 4ba7192adfca4e73bb9cca620676f6cf P 6590438bffac4f5d8023294a6d2fd4ff [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 05:02:28.091284 17313 raft_consensus.cc:504] T 4ba7192adfca4e73bb9cca620676f6cf P 6590438bffac4f5d8023294a6d2fd4ff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cc0328540d4a9db537a475b14a4ab3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45917 } } peers { permanent_uuid: "6590438bffac4f5d8023294a6d2fd4ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33823 } } peers { permanent_uuid: "1d219c86d9ea4b6aa6d894eb20867698" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42831 } }
I20260924 05:02:28.091444 17313 leader_election.cc:288] T 4ba7192adfca4e73bb9cca620676f6cf P 6590438bffac4f5d8023294a6d2fd4ff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e0cc0328540d4a9db537a475b14a4ab3 (127.0.98.3:45917), 1d219c86d9ea4b6aa6d894eb20867698 (127.0.98.2:42831)
W20260924 05:02:28.091854 16918 leader_election.cc:341] T 4ba7192adfca4e73bb9cca620676f6cf P 6590438bffac4f5d8023294a6d2fd4ff [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1d219c86d9ea4b6aa6d894eb20867698 (127.0.98.2:42831): Not found: Tablet not found: 4ba7192adfca4e73bb9cca620676f6cf
W20260924 05:02:28.091956 16919 leader_election.cc:341] T 4ba7192adfca4e73bb9cca620676f6cf P 6590438bffac4f5d8023294a6d2fd4ff [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e0cc0328540d4a9db537a475b14a4ab3 (127.0.98.3:45917): Not found: Tablet not found: 4ba7192adfca4e73bb9cca620676f6cf
I20260924 05:02:28.092020 16919 leader_election.cc:302] T 4ba7192adfca4e73bb9cca620676f6cf P 6590438bffac4f5d8023294a6d2fd4ff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6590438bffac4f5d8023294a6d2fd4ff; no voters: 1d219c86d9ea4b6aa6d894eb20867698, e0cc0328540d4a9db537a475b14a4ab3
I20260924 05:02:28.092096 17313 raft_consensus.cc:2726] T 4ba7192adfca4e73bb9cca620676f6cf P 6590438bffac4f5d8023294a6d2fd4ff [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 05:02:28.858536 17314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:28.858314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41625 (local address 127.0.98.1:49157)
0924 05:02:28.858448 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:28.858488 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":60,"threads_started":1}
I20260924 05:02:28.928975 17315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:28.928764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41625 (local address 127.0.98.2:43305)
0924 05:02:28.928895 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:28.928934 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":57,"threads_started":1}
I20260924 05:02:29.055460 17316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:29.055216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41625 (local address 127.0.98.3:48909)
0924 05:02:29.055360 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:29.055399 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":84,"threads_started":1}
I20260924 05:02:29.859185 17317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:29.858992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41625 (local address 127.0.98.1:37183)
0924 05:02:29.859107 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:29.859141 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":66,"threads_started":1}
I20260924 05:02:29.929595 17318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:29.929388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41625 (local address 127.0.98.2:37879)
0924 05:02:29.929523 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:29.929556 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":83,"threads_started":1}
I20260924 05:02:30.056150 17319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:30.055903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41625 (local address 127.0.98.3:50851)
0924 05:02:30.056060 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:30.056099 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":79,"threads_started":1}
I20260924 05:02:30.859841 17320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:30.859622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41625 (local address 127.0.98.1:41367)
0924 05:02:30.859756 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:30.859795 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":77,"threads_started":1}
I20260924 05:02:30.930181 17321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:30.929990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41625 (local address 127.0.98.2:49767)
0924 05:02:30.930102 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:30.930139 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":57,"threads_started":1}
I20260924 05:02:31.056861 17322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:31.056626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41625 (local address 127.0.98.3:37605)
0924 05:02:31.056765 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:31.056809 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":76,"threads_started":1}
I20260924 05:02:31.154922 17323 raft_consensus.cc:482] T 4ba7192adfca4e73bb9cca620676f6cf P 6590438bffac4f5d8023294a6d2fd4ff [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 05:02:31.155035 17323 raft_consensus.cc:504] T 4ba7192adfca4e73bb9cca620676f6cf P 6590438bffac4f5d8023294a6d2fd4ff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cc0328540d4a9db537a475b14a4ab3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45917 } } peers { permanent_uuid: "6590438bffac4f5d8023294a6d2fd4ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33823 } } peers { permanent_uuid: "1d219c86d9ea4b6aa6d894eb20867698" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42831 } }
I20260924 05:02:31.155205 17323 leader_election.cc:288] T 4ba7192adfca4e73bb9cca620676f6cf P 6590438bffac4f5d8023294a6d2fd4ff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e0cc0328540d4a9db537a475b14a4ab3 (127.0.98.3:45917), 1d219c86d9ea4b6aa6d894eb20867698 (127.0.98.2:42831)
W20260924 05:02:31.155612 16919 leader_election.cc:341] T 4ba7192adfca4e73bb9cca620676f6cf P 6590438bffac4f5d8023294a6d2fd4ff [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e0cc0328540d4a9db537a475b14a4ab3 (127.0.98.3:45917): Not found: Tablet not found: 4ba7192adfca4e73bb9cca620676f6cf
W20260924 05:02:31.155700 16918 leader_election.cc:341] T 4ba7192adfca4e73bb9cca620676f6cf P 6590438bffac4f5d8023294a6d2fd4ff [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1d219c86d9ea4b6aa6d894eb20867698 (127.0.98.2:42831): Not found: Tablet not found: 4ba7192adfca4e73bb9cca620676f6cf
I20260924 05:02:31.155743 16918 leader_election.cc:302] T 4ba7192adfca4e73bb9cca620676f6cf P 6590438bffac4f5d8023294a6d2fd4ff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6590438bffac4f5d8023294a6d2fd4ff; no voters: 1d219c86d9ea4b6aa6d894eb20867698, e0cc0328540d4a9db537a475b14a4ab3
I20260924 05:02:31.155821 17323 raft_consensus.cc:2726] T 4ba7192adfca4e73bb9cca620676f6cf P 6590438bffac4f5d8023294a6d2fd4ff [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 05:02:31.860441 17324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:31.860233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41625 (local address 127.0.98.1:60259)
0924 05:02:31.860355 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:31.860390 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":58,"threads_started":1}
I20260924 05:02:31.930905 17325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:31.930670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41625 (local address 127.0.98.2:38693)
0924 05:02:31.930817 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:31.930860 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":73,"threads_started":1}
I20260924 05:02:32.057562 17326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:32.057336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41625 (local address 127.0.98.3:57009)
0924 05:02:32.057483 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:32.057515 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":102,"threads_started":1}
I20260924 05:02:32.861183 17327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:32.860955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41625 (local address 127.0.98.1:47175)
0924 05:02:32.861097 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:32.861136 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":77,"threads_started":1}
I20260924 05:02:32.931576 17328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:32.931367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41625 (local address 127.0.98.2:40597)
0924 05:02:32.931491 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:32.931533 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":65,"threads_started":1}
I20260924 05:02:33.058218 17329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:33.057991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41625 (local address 127.0.98.3:57971)
0924 05:02:33.058126 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:33.058167 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":59,"threads_started":1}
I20260924 05:02:33.861872 17330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:33.861669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41625 (local address 127.0.98.1:60667)
0924 05:02:33.861783 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:33.861821 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":59,"threads_started":1}
I20260924 05:02:33.932169 17331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:33.931973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41625 (local address 127.0.98.2:47337)
0924 05:02:33.932083 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:33.932122 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":60,"threads_started":1}
I20260924 05:02:34.058840 17332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:34.058623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41625 (local address 127.0.98.3:47797)
0924 05:02:34.058752 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:34.058793 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":58,"threads_started":1}
I20260924 05:02:34.862636 17333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:34.862409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41625 (local address 127.0.98.1:37999)
0924 05:02:34.862547 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:34.862584 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":81,"threads_started":1}
I20260924 05:02:34.932749 17334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:34.932541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41625 (local address 127.0.98.2:47695)
0924 05:02:34.932660 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:34.932700 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":57,"threads_started":1}
I20260924 05:02:35.059561 17335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:35.059338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41625 (local address 127.0.98.3:35993)
0924 05:02:35.059468 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:35.059513 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":74,"threads_started":1}
I20260924 05:02:35.679947 17336 raft_consensus.cc:482] T 4ba7192adfca4e73bb9cca620676f6cf P 6590438bffac4f5d8023294a6d2fd4ff [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 05:02:35.680051 17336 raft_consensus.cc:504] T 4ba7192adfca4e73bb9cca620676f6cf P 6590438bffac4f5d8023294a6d2fd4ff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0cc0328540d4a9db537a475b14a4ab3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45917 } } peers { permanent_uuid: "6590438bffac4f5d8023294a6d2fd4ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33823 } } peers { permanent_uuid: "1d219c86d9ea4b6aa6d894eb20867698" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42831 } }
I20260924 05:02:35.680194 17336 leader_election.cc:288] T 4ba7192adfca4e73bb9cca620676f6cf P 6590438bffac4f5d8023294a6d2fd4ff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e0cc0328540d4a9db537a475b14a4ab3 (127.0.98.3:45917), 1d219c86d9ea4b6aa6d894eb20867698 (127.0.98.2:42831)
W20260924 05:02:35.680697 16918 leader_election.cc:341] T 4ba7192adfca4e73bb9cca620676f6cf P 6590438bffac4f5d8023294a6d2fd4ff [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1d219c86d9ea4b6aa6d894eb20867698 (127.0.98.2:42831): Not found: Tablet not found: 4ba7192adfca4e73bb9cca620676f6cf
W20260924 05:02:35.680775 16919 leader_election.cc:341] T 4ba7192adfca4e73bb9cca620676f6cf P 6590438bffac4f5d8023294a6d2fd4ff [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e0cc0328540d4a9db537a475b14a4ab3 (127.0.98.3:45917): Not found: Tablet not found: 4ba7192adfca4e73bb9cca620676f6cf
I20260924 05:02:35.680811 16919 leader_election.cc:302] T 4ba7192adfca4e73bb9cca620676f6cf P 6590438bffac4f5d8023294a6d2fd4ff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6590438bffac4f5d8023294a6d2fd4ff; no voters: 1d219c86d9ea4b6aa6d894eb20867698, e0cc0328540d4a9db537a475b14a4ab3
I20260924 05:02:35.680902 17336 raft_consensus.cc:2726] T 4ba7192adfca4e73bb9cca620676f6cf P 6590438bffac4f5d8023294a6d2fd4ff [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 05:02:35.863384 17337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:35.863127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41625 (local address 127.0.98.1:50943)
0924 05:02:35.863295 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:35.863335 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":62,"threads_started":1}
I20260924 05:02:35.933472 17338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:35.933170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41625 (local address 127.0.98.2:43605)
0924 05:02:35.933338 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:35.933377 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":73,"threads_started":1}
I20260924 05:02:36.060225 17339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:36.060017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41625 (local address 127.0.98.3:52229)
0924 05:02:36.060144 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:36.060178 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":78,"threads_started":1}
I20260924 05:02:36.864127 17340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:36.863893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41625 (local address 127.0.98.1:36389)
0924 05:02:36.864032 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:36.864068 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":74,"threads_started":1}
I20260924 05:02:36.934095 17341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:36.933883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41625 (local address 127.0.98.2:55001)
0924 05:02:36.934010 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:36.934049 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":74,"threads_started":1}
I20260924 05:02:37.060931 17342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:37.060725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41625 (local address 127.0.98.3:39781)
0924 05:02:37.060850 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:37.060884 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":73,"threads_started":1}
I20260924 05:02:37.864832 17343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:37.864592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41625 (local address 127.0.98.1:40835)
0924 05:02:37.864747 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:37.864788 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":77,"threads_started":1}
I20260924 05:02:37.934710 17344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:37.934469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41625 (local address 127.0.98.2:39145)
0924 05:02:37.934611 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:37.934652 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":56,"threads_started":1}
I20260924 05:02:38.061592 17345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:38.061376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41625 (local address 127.0.98.3:35113)
0924 05:02:38.061508 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:38.061543 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":69,"threads_started":1}
I20260924 05:02:38.865525 17346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:38.865274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41625 (local address 127.0.98.1:55611)
0924 05:02:38.865405 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:38.865479 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":80,"threads_started":1}
I20260924 05:02:38.935313 17347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:38.935103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41625 (local address 127.0.98.2:56241)
0924 05:02:38.935233 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:38.935270 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":57,"threads_started":1}
I20260924 05:02:39.062175 17348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:39.061972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41625 (local address 127.0.98.3:36623)
0924 05:02:39.062093 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:39.062127 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":72,"threads_started":1}
I20260924 05:02:39.866201 17349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:39.866011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41625 (local address 127.0.98.1:41889)
0924 05:02:39.866123 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:39.866158 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":66,"threads_started":1}
I20260924 05:02:39.936018 17350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:39.935803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41625 (local address 127.0.98.2:41303)
0924 05:02:39.935932 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:39.935971 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":56,"threads_started":1}
I20260924 05:02:40.058112 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 16869
I20260924 05:02:40.062508 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 17035
I20260924 05:02:40.063482 17351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:40.062670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41625 (local address 127.0.98.3:44631)
0924 05:02:40.063387 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:40.063429 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":698,"thread_start_us":61,"threads_started":1}
I20260924 05:02:40.066290 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 17167
2026-09-24T05:02:40Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58072 ms)
[----------] 30 tests from SecurityITest (230224 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225929848825-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:02:40 dist-test-slave-70st krb5kdc[17354](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:02:40 dist-test-slave-70st krb5kdc[17354](info): set up 2 sockets
Sep 24 05:02:40 dist-test-slave-70st krb5kdc[17354](info): commencing operation
krb5kdc: starting...
W20260924 05:02:41.012866 392 mini_kdc.cc:121] Time spent starting KDC: real 0.926s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225929848825-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225929848825-392-0/krb5kdc/test-user.keytab.
Sep 24 05:02:41 dist-test-slave-70st krb5kdc[17354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226161, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:02:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:02:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 05:02:41.128559 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225929848825-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225929848825-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225929848825-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45109
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45109
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:02:41.207369 17367 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:02:41.207589 17367 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:02:41.207617 17367 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:02:41.208765 17367 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:02:41.208806 17367 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:02:41.208823 17367 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:02:41.208839 17367 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:02:41.208853 17367 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:02:41.210341 17367 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225929848825-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225929848825-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45109
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45109
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17367
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225929848825-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:02:41.210597 17367 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:02:41.210794 17367 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:02:41.215772 17372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:02:41.215787 17367 server_base.cc:1034] running on GCE node
W20260924 05:02:41.215935 17373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:02:41.216059 17375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:02:41.216289 17367 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:02:41.216574 17367 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:02:41.217710 17367 hybrid_clock.cc:648] HybridClock initialized: now 1790226161217699 us; error 33 us; skew 500 ppm
Sep 24 05:02:41 dist-test-slave-70st krb5kdc[17354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226161, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:02:41.220121 17367 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 05:02:41.220597 17367 webserver.cc:460] Webserver started at http://127.0.98.62:35623/ using document root <none> and password file <none>
I20260924 05:02:41.220781 17367 fs_manager.cc:359] Metadata directory not provided
I20260924 05:02:41.220819 17367 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:02:41.220980 17367 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:02:41.222046 17367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225929848825-392-0/minicluster-data/master-0/data/instance:
uuid: "abaa5b0777a141faba6a9ab022a76d0b"
format_stamp: "Formatted at 2026-09-24 05:02:41 on dist-test-slave-70st"
server_key: "621ba0c887c26c516d11fa553b5636cb"
server_key_iv: "efd21294d78ee0284b7ebde27b6e12bf"
server_key_version: "encryptionkey@0"
I20260924 05:02:41.222332 17367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225929848825-392-0/minicluster-data/master-0/wal/instance:
uuid: "abaa5b0777a141faba6a9ab022a76d0b"
format_stamp: "Formatted at 2026-09-24 05:02:41 on dist-test-slave-70st"
server_key: "621ba0c887c26c516d11fa553b5636cb"
server_key_iv: "efd21294d78ee0284b7ebde27b6e12bf"
server_key_version: "encryptionkey@0"
I20260924 05:02:41.223568 17367 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 05:02:41.224188 17382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:02:41.224344 17367 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:02:41.224428 17367 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225929848825-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225929848825-392-0/minicluster-data/master-0/wal
uuid: "abaa5b0777a141faba6a9ab022a76d0b"
format_stamp: "Formatted at 2026-09-24 05:02:41 on dist-test-slave-70st"
server_key: "621ba0c887c26c516d11fa553b5636cb"
server_key_iv: "efd21294d78ee0284b7ebde27b6e12bf"
server_key_version: "encryptionkey@0"
I20260924 05:02:41.224498 17367 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225929848825-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225929848825-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225929848825-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:02:41.233140 17367 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:02:41.234933 17367 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:02:41.235061 17367 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:02:41.238657 17367 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45109
I20260924 05:02:41.238703 17434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45109 every 8 connection(s)
I20260924 05:02:41.238996 17367 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:02:41.239765 17435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:02:41.242019 17435 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abaa5b0777a141faba6a9ab022a76d0b: Bootstrap starting.
I20260924 05:02:41.242518 17435 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P abaa5b0777a141faba6a9ab022a76d0b: Neither blocks nor log segments found. Creating new log.
I20260924 05:02:41.242734 17435 log.cc:824] T 00000000000000000000000000000000 P abaa5b0777a141faba6a9ab022a76d0b: Log is configured to *not* fsync() on all Append() calls
I20260924 05:02:41.242911 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 17367
I20260924 05:02:41.243022 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225929848825-392-0/minicluster-data/master-0/wal/instance
I20260924 05:02:41.243243 392 external_mini_cluster.cc:1428] Setting key 48318ae2ade8467b473bd07f117c1ce1
I20260924 05:02:41.243559 17435 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abaa5b0777a141faba6a9ab022a76d0b: No bootstrap required, opened a new log
I20260924 05:02:41.245270 17435 raft_consensus.cc:348] T 00000000000000000000000000000000 P abaa5b0777a141faba6a9ab022a76d0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abaa5b0777a141faba6a9ab022a76d0b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45109 } }
I20260924 05:02:41.245410 17435 raft_consensus.cc:374] T 00000000000000000000000000000000 P abaa5b0777a141faba6a9ab022a76d0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:02:41.245508 17435 raft_consensus.cc:729] T 00000000000000000000000000000000 P abaa5b0777a141faba6a9ab022a76d0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abaa5b0777a141faba6a9ab022a76d0b, State: Initialized, Role: FOLLOWER
I20260924 05:02:41.245645 17435 consensus_queue.cc:260] T 00000000000000000000000000000000 P abaa5b0777a141faba6a9ab022a76d0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abaa5b0777a141faba6a9ab022a76d0b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45109 } }
I20260924 05:02:41.245725 17435 raft_consensus.cc:388] T 00000000000000000000000000000000 P abaa5b0777a141faba6a9ab022a76d0b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:02:41.245764 17435 raft_consensus.cc:482] T 00000000000000000000000000000000 P abaa5b0777a141faba6a9ab022a76d0b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:02:41.245807 17435 raft_consensus.cc:3037] T 00000000000000000000000000000000 P abaa5b0777a141faba6a9ab022a76d0b [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:02:41.246693 17435 raft_consensus.cc:504] T 00000000000000000000000000000000 P abaa5b0777a141faba6a9ab022a76d0b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abaa5b0777a141faba6a9ab022a76d0b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45109 } }
I20260924 05:02:41.246781 17435 leader_election.cc:302] T 00000000000000000000000000000000 P abaa5b0777a141faba6a9ab022a76d0b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: abaa5b0777a141faba6a9ab022a76d0b; no voters:
I20260924 05:02:41.246932 17435 leader_election.cc:288] T 00000000000000000000000000000000 P abaa5b0777a141faba6a9ab022a76d0b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:02:41.246971 17440 raft_consensus.cc:2781] T 00000000000000000000000000000000 P abaa5b0777a141faba6a9ab022a76d0b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:02:41.247118 17440 raft_consensus.cc:686] T 00000000000000000000000000000000 P abaa5b0777a141faba6a9ab022a76d0b [term 1 LEADER]: Becoming Leader. State: Replica: abaa5b0777a141faba6a9ab022a76d0b, State: Running, Role: LEADER
I20260924 05:02:41.247166 17435 sys_catalog.cc:564] T 00000000000000000000000000000000 P abaa5b0777a141faba6a9ab022a76d0b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:02:41.247198 17440 consensus_queue.cc:237] T 00000000000000000000000000000000 P abaa5b0777a141faba6a9ab022a76d0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abaa5b0777a141faba6a9ab022a76d0b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45109 } }
I20260924 05:02:41.247507 17440 sys_catalog.cc:455] T 00000000000000000000000000000000 P abaa5b0777a141faba6a9ab022a76d0b [sys.catalog]: SysCatalogTable state changed. Reason: New leader abaa5b0777a141faba6a9ab022a76d0b. Latest consensus state: current_term: 1 leader_uuid: "abaa5b0777a141faba6a9ab022a76d0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abaa5b0777a141faba6a9ab022a76d0b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45109 } } }
I20260924 05:02:41.247530 17441 sys_catalog.cc:455] T 00000000000000000000000000000000 P abaa5b0777a141faba6a9ab022a76d0b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "abaa5b0777a141faba6a9ab022a76d0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abaa5b0777a141faba6a9ab022a76d0b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45109 } } }
I20260924 05:02:41.247586 17440 sys_catalog.cc:458] T 00000000000000000000000000000000 P abaa5b0777a141faba6a9ab022a76d0b [sys.catalog]: This master's current role is: LEADER
I20260924 05:02:41.247592 17441 sys_catalog.cc:458] T 00000000000000000000000000000000 P abaa5b0777a141faba6a9ab022a76d0b [sys.catalog]: This master's current role is: LEADER
I20260924 05:02:41.247764 17445 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:02:41.248257 17445 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:02:41.249907 17445 catalog_manager.cc:1347] Generated new cluster ID: 4b383994c08749599935ea131e2e58ac
I20260924 05:02:41.249957 17445 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 05:02:41 dist-test-slave-70st krb5kdc[17354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226161, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:02:41.255285 17439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:41.243712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52838 (local address 127.0.98.62:45109)
0924 05:02:41.243889 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 05:02:41.243900 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 05:02:41.243941 (+ 41us) server_negotiation.cc:407] Connection header received
0924 05:02:41.245252 (+ 1311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:02:41.245259 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:02:41.245339 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:02:41.245513 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:02:41.246233 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:41.247365 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:41.248445 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:41.249037 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:41.249145 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:41.251818 (+ 2673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:02:41.251822 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:02:41.251822 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:02:41.251833 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:02:41.253721 (+ 1888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:02:41.254078 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:02:41.254082 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:02:41.254083 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:02:41.254207 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:02:41.254385 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:02:41.254386 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:02:41.254386 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:02:41.254678 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:02:41.254845 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0924 05:02:41.254912 (+ 67us) server_negotiation.cc:299] Negotiation successful
0924 05:02:41.255073 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":80,"threads_started":1}
I20260924 05:02:41.282800 17445 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:02:41.283337 17445 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:02:41.288345 17445 catalog_manager.cc:5868] T 00000000000000000000000000000000 P abaa5b0777a141faba6a9ab022a76d0b: Generated new TSK 0
I20260924 05:02:41.288529 17445 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225929848825-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225929848825-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 05:02:41.320138 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225929848825-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225929848825-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225929848825-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225929848825-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45109
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225929848825-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:02:41.423359 17461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:02:41.423550 17461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:02:41.423573 17461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:02:41.424633 17461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:02:41.424664 17461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:02:41.424687 17461 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 05:02:41.426151 17461 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225929848825-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225929848825-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225929848825-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225929848825-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17461
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225929848825-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:02:41.426365 17461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:02:41.426569 17461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:02:41.431524 17466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:02:41.431529 17469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:02:41.431648 17467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:02:41.431663 17461 server_base.cc:1034] running on GCE node
I20260924 05:02:41.431988 17461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:02:41.432219 17461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:02:41.433410 17461 hybrid_clock.cc:648] HybridClock initialized: now 1790226161433396 us; error 30 us; skew 500 ppm
Sep 24 05:02:41 dist-test-slave-70st krb5kdc[17354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226161, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:02:41.435909 17461 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 05:02:41.436323 17461 webserver.cc:460] Webserver started at http://127.0.98.1:33165/ using document root <none> and password file <none>
I20260924 05:02:41.436530 17461 fs_manager.cc:359] Metadata directory not provided
I20260924 05:02:41.436578 17461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:02:41.436697 17461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:02:41.437765 17461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225929848825-392-0/minicluster-data/ts-0/data/instance:
uuid: "d6b7e69b54b847869aa9dddfbd43a89d"
format_stamp: "Formatted at 2026-09-24 05:02:41 on dist-test-slave-70st"
server_key: "760291d1630bb944031b0578655ffd79"
server_key_iv: "a05d802e90df718818e046b1808a0a9b"
server_key_version: "encryptionkey@0"
I20260924 05:02:41.438078 17461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225929848825-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d6b7e69b54b847869aa9dddfbd43a89d"
format_stamp: "Formatted at 2026-09-24 05:02:41 on dist-test-slave-70st"
server_key: "760291d1630bb944031b0578655ffd79"
server_key_iv: "a05d802e90df718818e046b1808a0a9b"
server_key_version: "encryptionkey@0"
I20260924 05:02:41.439249 17461 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 05:02:41.440021 17476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:02:41.440202 17461 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 05:02:41.440284 17461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225929848825-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225929848825-392-0/minicluster-data/ts-0/wal
uuid: "d6b7e69b54b847869aa9dddfbd43a89d"
format_stamp: "Formatted at 2026-09-24 05:02:41 on dist-test-slave-70st"
server_key: "760291d1630bb944031b0578655ffd79"
server_key_iv: "a05d802e90df718818e046b1808a0a9b"
server_key_version: "encryptionkey@0"
I20260924 05:02:41.440372 17461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225929848825-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225929848825-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225929848825-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:02:41.450129 17461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:02:41.452284 17461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:02:41.452430 17461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:02:41.452636 17461 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:02:41.452935 17461 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:02:41.452968 17461 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:02:41.452997 17461 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:02:41.453014 17461 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:02:41.458369 17461 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41509
I20260924 05:02:41.458432 17589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41509 every 8 connection(s)
I20260924 05:02:41.458717 17461 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225929848825-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 05:02:41 dist-test-slave-70st krb5kdc[17354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226161, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:02:41.465390 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 17461
I20260924 05:02:41.465535 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225929848825-392-0/minicluster-data/ts-0/wal/instance
I20260924 05:02:41.465739 392 external_mini_cluster.cc:1428] Setting key 5c28bbfb4921936e29312f524f75d753
I20260924 05:02:41.467532 17439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:41.459425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49265 (local address 127.0.98.62:45109)
0924 05:02:41.459493 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 05:02:41.459495 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:02:41.459670 (+ 175us) server_negotiation.cc:407] Connection header received
0924 05:02:41.460552 (+ 882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:02:41.460553 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:02:41.460565 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:02:41.460623 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:02:41.461200 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:41.461749 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:41.462718 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:41.462910 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:41.462957 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:41.464923 (+ 1966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:02:41.464924 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:02:41.464925 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:02:41.464933 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:02:41.466367 (+ 1434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:02:41.466719 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:02:41.466721 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:02:41.466721 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:02:41.466781 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:02:41.466982 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:02:41.466983 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:02:41.466983 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:02:41.467099 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:02:41.467162 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 05:02:41.467363 (+ 201us) server_negotiation.cc:299] Negotiation successful
0924 05:02:41.467450 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 05:02:41.467662 17592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:41.459402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45109 (local address 127.0.98.1:49265)
0924 05:02:41.459546 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:41.459589 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 05:02:41.460468 (+ 879us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:02:41.460681 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:02:41.460684 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:02:41.460931 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:02:41.461122 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:41.461126 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:41.461809 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:41.461810 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:41.462651 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:41.462654 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:41.462980 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:41.462981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:41.463092 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:41.463581 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:02:41.463589 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:02:41.464835 (+ 1246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:02:41.466447 (+ 1612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:02:41.466455 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:02:41.466457 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:02:41.466657 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:02:41.466844 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:02:41.466844 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:02:41.466844 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:02:41.466934 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:02:41.467153 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:02:41.467155 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:02:41.467261 (+ 106us) client_negotiation.cc:769] Sending connection context
0924 05:02:41.467340 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 05:02:41.467450 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":66,"threads_started":1}
I20260924 05:02:41.468194 17590 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45109
I20260924 05:02:41.468277 17590 heartbeater.cc:464] Registering TS with master...
I20260924 05:02:41.468443 17590 heartbeater.cc:511] Master 127.0.98.62:45109 requested a full tablet report, sending...
I20260924 05:02:41.468935 17399 ts_manager.cc:194] Registered new tserver with Master: d6b7e69b54b847869aa9dddfbd43a89d (127.0.98.1:41509)
I20260924 05:02:41.469556 17399 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:49265
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225929848825-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225929848825-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 05:02:41.488777 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225929848825-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225929848825-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225929848825-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225929848825-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45109
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225929848825-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:02:41.566985 17595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:02:41.567176 17595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:02:41.567207 17595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:02:41.568324 17595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:02:41.568365 17595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:02:41.568399 17595 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 05:02:41.569828 17595 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225929848825-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225929848825-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225929848825-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225929848825-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17595
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225929848825-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:02:41.570042 17595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:02:41.570233 17595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:02:41.575265 17603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:02:41.575271 17601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:02:41.575474 17600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:02:41.575965 17595 server_base.cc:1034] running on GCE node
I20260924 05:02:41.576200 17595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:02:41.576471 17595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:02:41.577623 17595 hybrid_clock.cc:648] HybridClock initialized: now 1790226161577612 us; error 27 us; skew 500 ppm
Sep 24 05:02:41 dist-test-slave-70st krb5kdc[17354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226161, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:02:41.580009 17595 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 05:02:41.580431 17595 webserver.cc:460] Webserver started at http://127.0.98.2:45015/ using document root <none> and password file <none>
I20260924 05:02:41.580639 17595 fs_manager.cc:359] Metadata directory not provided
I20260924 05:02:41.580683 17595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:02:41.580790 17595 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:02:41.581832 17595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225929848825-392-0/minicluster-data/ts-1/data/instance:
uuid: "f3aac06e9ab24aaba701f7f4ff7ea29d"
format_stamp: "Formatted at 2026-09-24 05:02:41 on dist-test-slave-70st"
server_key: "e932a156514743b160bdfe6eeb1c7f06"
server_key_iv: "2703a4291d077dd498d425d93c01bd47"
server_key_version: "encryptionkey@0"
I20260924 05:02:41.582146 17595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225929848825-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f3aac06e9ab24aaba701f7f4ff7ea29d"
format_stamp: "Formatted at 2026-09-24 05:02:41 on dist-test-slave-70st"
server_key: "e932a156514743b160bdfe6eeb1c7f06"
server_key_iv: "2703a4291d077dd498d425d93c01bd47"
server_key_version: "encryptionkey@0"
I20260924 05:02:41.583395 17595 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 05:02:41.584112 17610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:02:41.584355 17595 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 05:02:41.584447 17595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225929848825-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225929848825-392-0/minicluster-data/ts-1/wal
uuid: "f3aac06e9ab24aaba701f7f4ff7ea29d"
format_stamp: "Formatted at 2026-09-24 05:02:41 on dist-test-slave-70st"
server_key: "e932a156514743b160bdfe6eeb1c7f06"
server_key_iv: "2703a4291d077dd498d425d93c01bd47"
server_key_version: "encryptionkey@0"
I20260924 05:02:41.584519 17595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225929848825-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225929848825-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225929848825-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:02:41.610821 17595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:02:41.612811 17595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:02:41.612951 17595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:02:41.613204 17595 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:02:41.613591 17595 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:02:41.613631 17595 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:02:41.613656 17595 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:02:41.613670 17595 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:02:41.619645 17595 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34927
I20260924 05:02:41.619689 17723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34927 every 8 connection(s)
I20260924 05:02:41.620002 17595 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225929848825-392-0/minicluster-data/ts-1/data/info.pb
I20260924 05:02:41.623437 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 17595
I20260924 05:02:41.623538 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225929848825-392-0/minicluster-data/ts-1/wal/instance
I20260924 05:02:41.623703 392 external_mini_cluster.cc:1428] Setting key c3188b7c7b6d699b4a97d444c136552c
Sep 24 05:02:41 dist-test-slave-70st krb5kdc[17354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226161, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:02:41.629630 17439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:41.620770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51021 (local address 127.0.98.62:45109)
0924 05:02:41.620851 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0924 05:02:41.620853 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:02:41.621025 (+ 172us) server_negotiation.cc:407] Connection header received
0924 05:02:41.621980 (+ 955us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:02:41.621981 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:02:41.621991 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:02:41.622048 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:02:41.622646 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:41.623139 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:41.624391 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:41.624583 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:41.624634 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:41.626915 (+ 2281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:02:41.626917 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:02:41.626917 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:02:41.626923 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:02:41.628275 (+ 1352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:02:41.628717 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:02:41.628718 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:02:41.628718 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:02:41.628771 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:02:41.629024 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:02:41.629025 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:02:41.629025 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:02:41.629120 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:02:41.629162 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0924 05:02:41.629500 (+ 338us) server_negotiation.cc:299] Negotiation successful
0924 05:02:41.629566 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 05:02:41.629876 17726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:41.620736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45109 (local address 127.0.98.2:51021)
0924 05:02:41.620891 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:41.620934 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 05:02:41.621884 (+ 950us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:02:41.622106 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:02:41.622109 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:02:41.622378 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:02:41.622574 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:41.622578 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:41.623237 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:41.623238 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:41.624282 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:41.624285 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:41.624659 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:41.624660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:41.624813 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:41.625339 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:02:41.625349 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:02:41.626834 (+ 1485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:02:41.628345 (+ 1511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:02:41.628352 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:02:41.628354 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:02:41.628641 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:02:41.628828 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:02:41.628829 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:02:41.628829 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:02:41.628964 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:02:41.629169 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:02:41.629175 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:02:41.629352 (+ 177us) client_negotiation.cc:769] Sending connection context
0924 05:02:41.629487 (+ 135us) client_negotiation.cc:240] Negotiation successful
0924 05:02:41.629650 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":83,"threads_started":1}
I20260924 05:02:41.630326 17724 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45109
I20260924 05:02:41.630415 17724 heartbeater.cc:464] Registering TS with master...
I20260924 05:02:41.630589 17724 heartbeater.cc:511] Master 127.0.98.62:45109 requested a full tablet report, sending...
I20260924 05:02:41.630923 17399 ts_manager.cc:194] Registered new tserver with Master: f3aac06e9ab24aaba701f7f4ff7ea29d (127.0.98.2:34927)
I20260924 05:02:41.631270 17399 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:51021
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225929848825-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225929848825-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 05:02:41.648828 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225929848825-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225929848825-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225929848825-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225929848825-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45109
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225929848825-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:02:41.727165 17729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:02:41.727372 17729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:02:41.727393 17729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:02:41.728561 17729 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:02:41.728605 17729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:02:41.728631 17729 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 05:02:41.730113 17729 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225929848825-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225929848825-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225929848825-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225929848825-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17729
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225929848825-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:02:41.730302 17729 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:02:41.730481 17729 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:02:41.735124 17734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:02:41.735091 17737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:02:41.735320 17729 server_base.cc:1034] running on GCE node
W20260924 05:02:41.735092 17735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:02:41.735579 17729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:02:41.735805 17729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:02:41.736950 17729 hybrid_clock.cc:648] HybridClock initialized: now 1790226161736951 us; error 35 us; skew 500 ppm
Sep 24 05:02:41 dist-test-slave-70st krb5kdc[17354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226161, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:02:41.739390 17729 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 05:02:41.739818 17729 webserver.cc:460] Webserver started at http://127.0.98.3:35269/ using document root <none> and password file <none>
I20260924 05:02:41.740012 17729 fs_manager.cc:359] Metadata directory not provided
I20260924 05:02:41.740059 17729 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:02:41.740176 17729 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:02:41.741163 17729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225929848825-392-0/minicluster-data/ts-2/data/instance:
uuid: "0547c0fa87154077bd2ac8a970d2939e"
format_stamp: "Formatted at 2026-09-24 05:02:41 on dist-test-slave-70st"
server_key: "18ddc786c5b91c0fa1f662e8dfff32e6"
server_key_iv: "b215baececed5ce327e3ed39500040a8"
server_key_version: "encryptionkey@0"
I20260924 05:02:41.741531 17729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225929848825-392-0/minicluster-data/ts-2/wal/instance:
uuid: "0547c0fa87154077bd2ac8a970d2939e"
format_stamp: "Formatted at 2026-09-24 05:02:41 on dist-test-slave-70st"
server_key: "18ddc786c5b91c0fa1f662e8dfff32e6"
server_key_iv: "b215baececed5ce327e3ed39500040a8"
server_key_version: "encryptionkey@0"
I20260924 05:02:41.742805 17729 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 05:02:41.743505 17744 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:02:41.743661 17729 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:02:41.743746 17729 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225929848825-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225929848825-392-0/minicluster-data/ts-2/wal
uuid: "0547c0fa87154077bd2ac8a970d2939e"
format_stamp: "Formatted at 2026-09-24 05:02:41 on dist-test-slave-70st"
server_key: "18ddc786c5b91c0fa1f662e8dfff32e6"
server_key_iv: "b215baececed5ce327e3ed39500040a8"
server_key_version: "encryptionkey@0"
I20260924 05:02:41.743813 17729 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225929848825-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225929848825-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225929848825-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:02:41.755084 17729 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:02:41.757125 17729 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:02:41.757259 17729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:02:41.757499 17729 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:02:41.757786 17729 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:02:41.757819 17729 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:02:41.757848 17729 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:02:41.757870 17729 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:02:41.763285 17729 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39275
I20260924 05:02:41.763320 17857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39275 every 8 connection(s)
I20260924 05:02:41.763617 17729 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225929848825-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 05:02:41 dist-test-slave-70st krb5kdc[17354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226161, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:02:41.771963 17439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:41.764244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49123 (local address 127.0.98.62:45109)
0924 05:02:41.764288 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 05:02:41.764290 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:02:41.764493 (+ 203us) server_negotiation.cc:407] Connection header received
0924 05:02:41.765385 (+ 892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:02:41.765387 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:02:41.765398 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:02:41.765478 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:02:41.766052 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:41.766543 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:41.767496 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:41.767677 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:41.767724 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:41.769600 (+ 1876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:02:41.769601 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:02:41.769602 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:02:41.769608 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:02:41.770802 (+ 1194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:02:41.771137 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:02:41.771138 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:02:41.771138 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:02:41.771182 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:02:41.771391 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:02:41.771392 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:02:41.771392 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:02:41.771533 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:02:41.771593 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 05:02:41.771834 (+ 241us) server_negotiation.cc:299] Negotiation successful
0924 05:02:41.771897 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 05:02:41.772105 17860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:41.764214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45109 (local address 127.0.98.3:49123)
0924 05:02:41.764388 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:41.764422 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 05:02:41.765287 (+ 865us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:02:41.765548 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:02:41.765551 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:02:41.765796 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:02:41.765976 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:41.765980 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:41.766620 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:41.766620 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:41.767428 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:41.767430 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:41.767732 (+ 302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:41.767733 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:41.767820 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:41.768218 (+ 398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:02:41.768226 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:02:41.769530 (+ 1304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:02:41.770875 (+ 1345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:02:41.770880 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:02:41.770882 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:02:41.771081 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:02:41.771225 (+ 144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:02:41.771225 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:02:41.771226 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:02:41.771311 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:02:41.771602 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:02:41.771607 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:02:41.771729 (+ 122us) client_negotiation.cc:769] Sending connection context
0924 05:02:41.771813 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 05:02:41.771928 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":87,"threads_started":1}
I20260924 05:02:41.772554 17858 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45109
I20260924 05:02:41.772632 17858 heartbeater.cc:464] Registering TS with master...
I20260924 05:02:41.772778 17858 heartbeater.cc:511] Master 127.0.98.62:45109 requested a full tablet report, sending...
I20260924 05:02:41.773114 17399 ts_manager.cc:194] Registered new tserver with Master: 0547c0fa87154077bd2ac8a970d2939e (127.0.98.3:39275)
I20260924 05:02:41.773157 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 17729
I20260924 05:02:41.773252 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225929848825-392-0/minicluster-data/ts-2/wal/instance
I20260924 05:02:41.773466 392 external_mini_cluster.cc:1428] Setting key 32f7edacef9336258bdc48c2f5d518cc
I20260924 05:02:41.773598 17399 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:49123
I20260924 05:02:41.774196 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 05:02:41.780542 17439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:41.775434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52848 (local address 127.0.98.62:45109)
0924 05:02:41.775483 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 05:02:41.775485 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:02:41.775574 (+ 89us) server_negotiation.cc:407] Connection header received
0924 05:02:41.775614 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:02:41.775614 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:02:41.775622 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:02:41.775667 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:02:41.776115 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:41.776571 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:41.777278 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:41.777511 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:41.777560 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:41.778559 (+ 999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:02:41.778560 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:02:41.778560 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:02:41.778566 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:02:41.779722 (+ 1156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:02:41.780003 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:02:41.780004 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:02:41.780004 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:02:41.780046 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:02:41.780160 (+ 114us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:02:41.780160 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:02:41.780160 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:02:41.780278 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:02:41.780358 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 05:02:41.780431 (+ 73us) server_negotiation.cc:299] Negotiation successful
0924 05:02:41.780483 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 05:02:41.781894 17399 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52848:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 05:02:41.783393 17399 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 05:02:41.789491 17866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:41.785395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39275 (local address 127.0.0.1:45952)
0924 05:02:41.785684 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:41.785694 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:02:41.785746 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:02:41.786027 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:02:41.786029 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:02:41.786031 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:02:41.786097 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:41.786099 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:41.787051 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:41.787052 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:41.788400 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:41.788402 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:41.789272 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:41.789273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:41.789336 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:41.789339 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:02:41.789381 (+ 42us) client_negotiation.cc:240] Negotiation successful
0924 05:02:41.789432 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":81,"threads_started":1}
I20260924 05:02:41.789534 17870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:41.785726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45952 (local address 127.0.98.3:39275)
0924 05:02:41.785881 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 05:02:41.785885 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 05:02:41.785895 (+ 10us) server_negotiation.cc:407] Connection header received
0924 05:02:41.785928 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:02:41.785929 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:02:41.785942 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:02:41.786004 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:02:41.786173 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:41.786975 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:41.788471 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:41.789169 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:41.789264 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:41.789315 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 05:02:41.789402 (+ 87us) server_negotiation.cc:299] Negotiation successful
0924 05:02:41.789455 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":77,"threads_started":1}
I20260924 05:02:41.789577 17871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:41.785739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56446 (local address 127.0.98.2:34927)
0924 05:02:41.785924 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 05:02:41.785929 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 05:02:41.786317 (+ 388us) server_negotiation.cc:407] Connection header received
0924 05:02:41.786364 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:02:41.786366 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:02:41.786381 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:02:41.786446 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:02:41.786537 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:41.787401 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:41.788496 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:41.789076 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:41.789136 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:41.789193 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 05:02:41.789466 (+ 273us) server_negotiation.cc:299] Negotiation successful
0924 05:02:41.789505 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":94,"threads_started":1}
I20260924 05:02:41.789680 17869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:41.785615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34927 (local address 127.0.0.1:56446)
0924 05:02:41.786203 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:41.786212 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:02:41.786257 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:02:41.786452 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:02:41.786453 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:02:41.786455 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:02:41.786490 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:41.786491 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:41.787486 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:41.787487 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:41.788435 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:41.788438 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:41.789148 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:41.789149 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:41.789225 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:41.789238 (+ 13us) client_negotiation.cc:769] Sending connection context
0924 05:02:41.789302 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 05:02:41.789356 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"mutex_wait_us":3,"thread_start_us":49,"threads_started":1}
I20260924 05:02:41.790267 17792 tablet_service.cc:1467] Processing CreateTablet for tablet a409478b3adc45d08c9a65fff7e2fa3e (DEFAULT_TABLE table=test-table [id=d04f1447cbc54f91b7fe968f50f16af6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:02:41.790371 17658 tablet_service.cc:1467] Processing CreateTablet for tablet a409478b3adc45d08c9a65fff7e2fa3e (DEFAULT_TABLE table=test-table [id=d04f1447cbc54f91b7fe968f50f16af6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:02:41.790586 17792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a409478b3adc45d08c9a65fff7e2fa3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:02:41.790683 17658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a409478b3adc45d08c9a65fff7e2fa3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:02:41.790987 17867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:41.785393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41509 (local address 127.0.0.1:47764)
0924 05:02:41.785540 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:41.785577 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 05:02:41.785637 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:02:41.786107 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:02:41.786108 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:02:41.786110 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:02:41.786159 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:41.786160 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:41.788635 (+ 2475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:41.788636 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:41.789694 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:41.789698 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:41.790760 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:41.790761 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:41.790822 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:41.790827 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 05:02:41.790874 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 05:02:41.790909 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":102,"threads_started":1}
I20260924 05:02:41.790998 17868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:41.785534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47764 (local address 127.0.98.1:41509)
0924 05:02:41.785966 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0924 05:02:41.785969 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:02:41.785977 (+ 8us) server_negotiation.cc:407] Connection header received
0924 05:02:41.786016 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:02:41.786018 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:02:41.786033 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:02:41.786112 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:02:41.787579 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:41.788553 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:41.789992 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:41.790658 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:41.790728 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:41.790783 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 05:02:41.790887 (+ 104us) server_negotiation.cc:299] Negotiation successful
0924 05:02:41.790927 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":72,"threads_started":1}
I20260924 05:02:41.791625 17524 tablet_service.cc:1467] Processing CreateTablet for tablet a409478b3adc45d08c9a65fff7e2fa3e (DEFAULT_TABLE table=test-table [id=d04f1447cbc54f91b7fe968f50f16af6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:02:41.791909 17524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a409478b3adc45d08c9a65fff7e2fa3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:02:41.793049 17872 tablet_bootstrap.cc:492] T a409478b3adc45d08c9a65fff7e2fa3e P 0547c0fa87154077bd2ac8a970d2939e: Bootstrap starting.
I20260924 05:02:41.793952 17872 tablet_bootstrap.cc:654] T a409478b3adc45d08c9a65fff7e2fa3e P 0547c0fa87154077bd2ac8a970d2939e: Neither blocks nor log segments found. Creating new log.
I20260924 05:02:41.794396 17872 log.cc:824] T a409478b3adc45d08c9a65fff7e2fa3e P 0547c0fa87154077bd2ac8a970d2939e: Log is configured to *not* fsync() on all Append() calls
I20260924 05:02:41.794641 17874 tablet_bootstrap.cc:492] T a409478b3adc45d08c9a65fff7e2fa3e P d6b7e69b54b847869aa9dddfbd43a89d: Bootstrap starting.
I20260924 05:02:41.794723 17873 tablet_bootstrap.cc:492] T a409478b3adc45d08c9a65fff7e2fa3e P f3aac06e9ab24aaba701f7f4ff7ea29d: Bootstrap starting.
I20260924 05:02:41.795143 17872 tablet_bootstrap.cc:492] T a409478b3adc45d08c9a65fff7e2fa3e P 0547c0fa87154077bd2ac8a970d2939e: No bootstrap required, opened a new log
I20260924 05:02:41.795215 17872 ts_tablet_manager.cc:1397] T a409478b3adc45d08c9a65fff7e2fa3e P 0547c0fa87154077bd2ac8a970d2939e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 05:02:41.795233 17874 tablet_bootstrap.cc:654] T a409478b3adc45d08c9a65fff7e2fa3e P d6b7e69b54b847869aa9dddfbd43a89d: Neither blocks nor log segments found. Creating new log.
I20260924 05:02:41.795358 17873 tablet_bootstrap.cc:654] T a409478b3adc45d08c9a65fff7e2fa3e P f3aac06e9ab24aaba701f7f4ff7ea29d: Neither blocks nor log segments found. Creating new log.
I20260924 05:02:41.795670 17874 log.cc:824] T a409478b3adc45d08c9a65fff7e2fa3e P d6b7e69b54b847869aa9dddfbd43a89d: Log is configured to *not* fsync() on all Append() calls
I20260924 05:02:41.795856 17873 log.cc:824] T a409478b3adc45d08c9a65fff7e2fa3e P f3aac06e9ab24aaba701f7f4ff7ea29d: Log is configured to *not* fsync() on all Append() calls
I20260924 05:02:41.796324 17874 tablet_bootstrap.cc:492] T a409478b3adc45d08c9a65fff7e2fa3e P d6b7e69b54b847869aa9dddfbd43a89d: No bootstrap required, opened a new log
I20260924 05:02:41.796386 17874 ts_tablet_manager.cc:1397] T a409478b3adc45d08c9a65fff7e2fa3e P d6b7e69b54b847869aa9dddfbd43a89d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 05:02:41.796602 17873 tablet_bootstrap.cc:492] T a409478b3adc45d08c9a65fff7e2fa3e P f3aac06e9ab24aaba701f7f4ff7ea29d: No bootstrap required, opened a new log
I20260924 05:02:41.796715 17873 ts_tablet_manager.cc:1397] T a409478b3adc45d08c9a65fff7e2fa3e P f3aac06e9ab24aaba701f7f4ff7ea29d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 05:02:41.797152 17872 raft_consensus.cc:348] T a409478b3adc45d08c9a65fff7e2fa3e P 0547c0fa87154077bd2ac8a970d2939e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b7e69b54b847869aa9dddfbd43a89d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41509 } } peers { permanent_uuid: "0547c0fa87154077bd2ac8a970d2939e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39275 } } peers { permanent_uuid: "f3aac06e9ab24aaba701f7f4ff7ea29d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34927 } }
I20260924 05:02:41.797326 17872 raft_consensus.cc:374] T a409478b3adc45d08c9a65fff7e2fa3e P 0547c0fa87154077bd2ac8a970d2939e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:02:41.797361 17872 raft_consensus.cc:729] T a409478b3adc45d08c9a65fff7e2fa3e P 0547c0fa87154077bd2ac8a970d2939e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0547c0fa87154077bd2ac8a970d2939e, State: Initialized, Role: FOLLOWER
I20260924 05:02:41.797534 17872 consensus_queue.cc:260] T a409478b3adc45d08c9a65fff7e2fa3e P 0547c0fa87154077bd2ac8a970d2939e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b7e69b54b847869aa9dddfbd43a89d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41509 } } peers { permanent_uuid: "0547c0fa87154077bd2ac8a970d2939e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39275 } } peers { permanent_uuid: "f3aac06e9ab24aaba701f7f4ff7ea29d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34927 } }
I20260924 05:02:41.797806 17872 ts_tablet_manager.cc:1428] T a409478b3adc45d08c9a65fff7e2fa3e P 0547c0fa87154077bd2ac8a970d2939e: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260924 05:02:41.797960 17858 heartbeater.cc:503] Master 127.0.98.62:45109 was elected leader, sending a full tablet report...
I20260924 05:02:41.797989 17874 raft_consensus.cc:348] T a409478b3adc45d08c9a65fff7e2fa3e P d6b7e69b54b847869aa9dddfbd43a89d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b7e69b54b847869aa9dddfbd43a89d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41509 } } peers { permanent_uuid: "0547c0fa87154077bd2ac8a970d2939e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39275 } } peers { permanent_uuid: "f3aac06e9ab24aaba701f7f4ff7ea29d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34927 } }
I20260924 05:02:41.798143 17874 raft_consensus.cc:374] T a409478b3adc45d08c9a65fff7e2fa3e P d6b7e69b54b847869aa9dddfbd43a89d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:02:41.798208 17874 raft_consensus.cc:729] T a409478b3adc45d08c9a65fff7e2fa3e P d6b7e69b54b847869aa9dddfbd43a89d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6b7e69b54b847869aa9dddfbd43a89d, State: Initialized, Role: FOLLOWER
I20260924 05:02:41.798362 17874 consensus_queue.cc:260] T a409478b3adc45d08c9a65fff7e2fa3e P d6b7e69b54b847869aa9dddfbd43a89d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b7e69b54b847869aa9dddfbd43a89d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41509 } } peers { permanent_uuid: "0547c0fa87154077bd2ac8a970d2939e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39275 } } peers { permanent_uuid: "f3aac06e9ab24aaba701f7f4ff7ea29d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34927 } }
I20260924 05:02:41.798604 17590 heartbeater.cc:503] Master 127.0.98.62:45109 was elected leader, sending a full tablet report...
I20260924 05:02:41.798657 17873 raft_consensus.cc:348] T a409478b3adc45d08c9a65fff7e2fa3e P f3aac06e9ab24aaba701f7f4ff7ea29d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b7e69b54b847869aa9dddfbd43a89d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41509 } } peers { permanent_uuid: "0547c0fa87154077bd2ac8a970d2939e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39275 } } peers { permanent_uuid: "f3aac06e9ab24aaba701f7f4ff7ea29d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34927 } }
I20260924 05:02:41.798818 17873 raft_consensus.cc:374] T a409478b3adc45d08c9a65fff7e2fa3e P f3aac06e9ab24aaba701f7f4ff7ea29d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:02:41.798853 17873 raft_consensus.cc:729] T a409478b3adc45d08c9a65fff7e2fa3e P f3aac06e9ab24aaba701f7f4ff7ea29d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3aac06e9ab24aaba701f7f4ff7ea29d, State: Initialized, Role: FOLLOWER
I20260924 05:02:41.798941 17874 ts_tablet_manager.cc:1428] T a409478b3adc45d08c9a65fff7e2fa3e P d6b7e69b54b847869aa9dddfbd43a89d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260924 05:02:41.799033 17873 consensus_queue.cc:260] T a409478b3adc45d08c9a65fff7e2fa3e P f3aac06e9ab24aaba701f7f4ff7ea29d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b7e69b54b847869aa9dddfbd43a89d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41509 } } peers { permanent_uuid: "0547c0fa87154077bd2ac8a970d2939e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39275 } } peers { permanent_uuid: "f3aac06e9ab24aaba701f7f4ff7ea29d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34927 } }
I20260924 05:02:41.799296 17724 heartbeater.cc:503] Master 127.0.98.62:45109 was elected leader, sending a full tablet report...
I20260924 05:02:41.799296 17873 ts_tablet_manager.cc:1428] T a409478b3adc45d08c9a65fff7e2fa3e P f3aac06e9ab24aaba701f7f4ff7ea29d: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
W20260924 05:02:41.870573 17725 tablet.cc:2369] T a409478b3adc45d08c9a65fff7e2fa3e P f3aac06e9ab24aaba701f7f4ff7ea29d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 05:02:41.895627 17880 raft_consensus.cc:482] T a409478b3adc45d08c9a65fff7e2fa3e P f3aac06e9ab24aaba701f7f4ff7ea29d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 05:02:41.895750 17880 raft_consensus.cc:504] T a409478b3adc45d08c9a65fff7e2fa3e P f3aac06e9ab24aaba701f7f4ff7ea29d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b7e69b54b847869aa9dddfbd43a89d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41509 } } peers { permanent_uuid: "0547c0fa87154077bd2ac8a970d2939e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39275 } } peers { permanent_uuid: "f3aac06e9ab24aaba701f7f4ff7ea29d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34927 } }
I20260924 05:02:41.896135 17880 leader_election.cc:288] T a409478b3adc45d08c9a65fff7e2fa3e P f3aac06e9ab24aaba701f7f4ff7ea29d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6b7e69b54b847869aa9dddfbd43a89d (127.0.98.1:41509), 0547c0fa87154077bd2ac8a970d2939e (127.0.98.3:39275)
I20260924 05:02:41.898799 17868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:41.896400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44073 (local address 127.0.98.1:41509)
0924 05:02:41.896457 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 05:02:41.896459 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:02:41.896468 (+ 9us) server_negotiation.cc:407] Connection header received
0924 05:02:41.896497 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:02:41.896498 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:02:41.896509 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:02:41.896555 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:02:41.896696 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:41.897160 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:41.898154 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:41.898508 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:41.898567 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:41.898578 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 05:02:41.898698 (+ 120us) server_negotiation.cc:299] Negotiation successful
0924 05:02:41.898733 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 05:02:41.898799 17726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:41.896322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41509 (local address 127.0.98.2:44073)
0924 05:02:41.896378 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:41.896390 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 05:02:41.896434 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:02:41.896564 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:02:41.896566 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:02:41.896567 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:02:41.896638 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:41.896639 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:41.897235 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:41.897236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:41.898072 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:41.898075 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:41.898574 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:41.898575 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:41.898640 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:41.898643 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:02:41.898688 (+ 45us) client_negotiation.cc:240] Negotiation successful
0924 05:02:41.898723 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":40}
I20260924 05:02:41.898885 17870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:41.896461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48751 (local address 127.0.98.3:39275)
0924 05:02:41.896515 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 05:02:41.896518 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:02:41.896590 (+ 72us) server_negotiation.cc:407] Connection header received
0924 05:02:41.896636 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:02:41.896637 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:02:41.896647 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:02:41.896688 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:02:41.896833 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:41.897277 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:41.898208 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:41.898568 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:41.898627 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:41.898637 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 05:02:41.898781 (+ 144us) server_negotiation.cc:299] Negotiation successful
0924 05:02:41.898815 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 05:02:41.898912 17881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:41.896331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39275 (local address 127.0.98.2:48751)
0924 05:02:41.896516 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:41.896525 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:02:41.896586 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:02:41.896700 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:02:41.896701 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:02:41.896702 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:02:41.896774 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:41.896775 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:41.897350 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:41.897351 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:41.898133 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:41.898135 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:41.898643 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:41.898644 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:41.898719 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:41.898722 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:02:41.898764 (+ 42us) client_negotiation.cc:240] Negotiation successful
0924 05:02:41.898798 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":105,"threads_started":1}
I20260924 05:02:41.899142 17544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a409478b3adc45d08c9a65fff7e2fa3e" candidate_uuid: "f3aac06e9ab24aaba701f7f4ff7ea29d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6b7e69b54b847869aa9dddfbd43a89d" is_pre_election: true
I20260924 05:02:41.899169 17812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a409478b3adc45d08c9a65fff7e2fa3e" candidate_uuid: "f3aac06e9ab24aaba701f7f4ff7ea29d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0547c0fa87154077bd2ac8a970d2939e" is_pre_election: true
I20260924 05:02:41.899291 17544 raft_consensus.cc:2454] T a409478b3adc45d08c9a65fff7e2fa3e P d6b7e69b54b847869aa9dddfbd43a89d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3aac06e9ab24aaba701f7f4ff7ea29d in term 0.
I20260924 05:02:41.899302 17812 raft_consensus.cc:2454] T a409478b3adc45d08c9a65fff7e2fa3e P 0547c0fa87154077bd2ac8a970d2939e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3aac06e9ab24aaba701f7f4ff7ea29d in term 0.
I20260924 05:02:41.899505 17613 leader_election.cc:302] T a409478b3adc45d08c9a65fff7e2fa3e P f3aac06e9ab24aaba701f7f4ff7ea29d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d6b7e69b54b847869aa9dddfbd43a89d, f3aac06e9ab24aaba701f7f4ff7ea29d; no voters:
I20260924 05:02:41.899664 17880 raft_consensus.cc:2781] T a409478b3adc45d08c9a65fff7e2fa3e P f3aac06e9ab24aaba701f7f4ff7ea29d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 05:02:41.899710 17880 raft_consensus.cc:482] T a409478b3adc45d08c9a65fff7e2fa3e P f3aac06e9ab24aaba701f7f4ff7ea29d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 05:02:41.899729 17880 raft_consensus.cc:3037] T a409478b3adc45d08c9a65fff7e2fa3e P f3aac06e9ab24aaba701f7f4ff7ea29d [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:02:41.900434 17880 raft_consensus.cc:504] T a409478b3adc45d08c9a65fff7e2fa3e P f3aac06e9ab24aaba701f7f4ff7ea29d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b7e69b54b847869aa9dddfbd43a89d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41509 } } peers { permanent_uuid: "0547c0fa87154077bd2ac8a970d2939e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39275 } } peers { permanent_uuid: "f3aac06e9ab24aaba701f7f4ff7ea29d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34927 } }
I20260924 05:02:41.900610 17880 leader_election.cc:288] T a409478b3adc45d08c9a65fff7e2fa3e P f3aac06e9ab24aaba701f7f4ff7ea29d [CANDIDATE]: Term 1 election: Requested vote from peers d6b7e69b54b847869aa9dddfbd43a89d (127.0.98.1:41509), 0547c0fa87154077bd2ac8a970d2939e (127.0.98.3:39275)
I20260924 05:02:41.900676 17812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a409478b3adc45d08c9a65fff7e2fa3e" candidate_uuid: "f3aac06e9ab24aaba701f7f4ff7ea29d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0547c0fa87154077bd2ac8a970d2939e"
I20260924 05:02:41.900720 17544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a409478b3adc45d08c9a65fff7e2fa3e" candidate_uuid: "f3aac06e9ab24aaba701f7f4ff7ea29d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6b7e69b54b847869aa9dddfbd43a89d"
I20260924 05:02:41.900755 17812 raft_consensus.cc:3037] T a409478b3adc45d08c9a65fff7e2fa3e P 0547c0fa87154077bd2ac8a970d2939e [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:02:41.900796 17544 raft_consensus.cc:3037] T a409478b3adc45d08c9a65fff7e2fa3e P d6b7e69b54b847869aa9dddfbd43a89d [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:02:41.901377 17812 raft_consensus.cc:2454] T a409478b3adc45d08c9a65fff7e2fa3e P 0547c0fa87154077bd2ac8a970d2939e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3aac06e9ab24aaba701f7f4ff7ea29d in term 1.
I20260924 05:02:41.901490 17544 raft_consensus.cc:2454] T a409478b3adc45d08c9a65fff7e2fa3e P d6b7e69b54b847869aa9dddfbd43a89d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3aac06e9ab24aaba701f7f4ff7ea29d in term 1.
I20260924 05:02:41.901556 17614 leader_election.cc:302] T a409478b3adc45d08c9a65fff7e2fa3e P f3aac06e9ab24aaba701f7f4ff7ea29d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0547c0fa87154077bd2ac8a970d2939e, f3aac06e9ab24aaba701f7f4ff7ea29d; no voters:
I20260924 05:02:41.901665 17880 raft_consensus.cc:2781] T a409478b3adc45d08c9a65fff7e2fa3e P f3aac06e9ab24aaba701f7f4ff7ea29d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:02:41.901805 17880 raft_consensus.cc:686] T a409478b3adc45d08c9a65fff7e2fa3e P f3aac06e9ab24aaba701f7f4ff7ea29d [term 1 LEADER]: Becoming Leader. State: Replica: f3aac06e9ab24aaba701f7f4ff7ea29d, State: Running, Role: LEADER
I20260924 05:02:41.901885 17880 consensus_queue.cc:237] T a409478b3adc45d08c9a65fff7e2fa3e P f3aac06e9ab24aaba701f7f4ff7ea29d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b7e69b54b847869aa9dddfbd43a89d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41509 } } peers { permanent_uuid: "0547c0fa87154077bd2ac8a970d2939e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39275 } } peers { permanent_uuid: "f3aac06e9ab24aaba701f7f4ff7ea29d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34927 } }
I20260924 05:02:41.902701 17399 catalog_manager.cc:5495] T a409478b3adc45d08c9a65fff7e2fa3e P f3aac06e9ab24aaba701f7f4ff7ea29d reported cstate change: term changed from 0 to 1, leader changed from <none> to f3aac06e9ab24aaba701f7f4ff7ea29d (127.0.98.2). New cstate: current_term: 1 leader_uuid: "f3aac06e9ab24aaba701f7f4ff7ea29d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b7e69b54b847869aa9dddfbd43a89d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0547c0fa87154077bd2ac8a970d2939e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39275 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f3aac06e9ab24aaba701f7f4ff7ea29d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34927 } health_report { overall_health: HEALTHY } } }
I20260924 05:02:41.932971 17871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:41.930813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56460 (local address 127.0.98.2:34927)
0924 05:02:41.930857 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 05:02:41.930859 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:02:41.930867 (+ 8us) server_negotiation.cc:407] Connection header received
0924 05:02:41.930901 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:02:41.930901 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:02:41.930921 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:02:41.930980 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:02:41.931079 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:41.931579 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:41.932348 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:41.932538 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:41.932591 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:41.932700 (+ 109us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:02:41.932803 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:02:41.932859 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 05:02:41.932876 (+ 17us) server_negotiation.cc:299] Negotiation successful
0924 05:02:41.932893 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 05:02:41.934755 17812 raft_consensus.cc:1261] T a409478b3adc45d08c9a65fff7e2fa3e P 0547c0fa87154077bd2ac8a970d2939e [term 1 FOLLOWER]: Refusing update from remote peer f3aac06e9ab24aaba701f7f4ff7ea29d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:02:41.934763 17544 raft_consensus.cc:1261] T a409478b3adc45d08c9a65fff7e2fa3e P d6b7e69b54b847869aa9dddfbd43a89d [term 1 FOLLOWER]: Refusing update from remote peer f3aac06e9ab24aaba701f7f4ff7ea29d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:02:41.935104 17882 consensus_queue.cc:1035] T a409478b3adc45d08c9a65fff7e2fa3e P f3aac06e9ab24aaba701f7f4ff7ea29d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0547c0fa87154077bd2ac8a970d2939e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39275 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:02:41.935225 17880 consensus_queue.cc:1035] T a409478b3adc45d08c9a65fff7e2fa3e P f3aac06e9ab24aaba701f7f4ff7ea29d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6b7e69b54b847869aa9dddfbd43a89d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:02:41.937369 17885 mvcc.cc:205] Tried to move back new op lower bound from 7332766359282249728 to 7332766359150661632. Current Snapshot: MvccSnapshot[applied={T|T < 7332766359282249728}]
I20260924 05:02:41.939755 17398 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52848:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:02:41.939837 17398 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52848:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:02:41.940546 17398 catalog_manager.cc:5782] T 00000000000000000000000000000000 P abaa5b0777a141faba6a9ab022a76d0b: Sending DeleteTablet for 3 replicas of tablet a409478b3adc45d08c9a65fff7e2fa3e
I20260924 05:02:41.940824 17524 tablet_service.cc:1514] Processing DeleteTablet for tablet a409478b3adc45d08c9a65fff7e2fa3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:02:41 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47764
I20260924 05:02:41.940874 17792 tablet_service.cc:1514] Processing DeleteTablet for tablet a409478b3adc45d08c9a65fff7e2fa3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:02:41 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45952
I20260924 05:02:41.940902 17658 tablet_service.cc:1514] Processing DeleteTablet for tablet a409478b3adc45d08c9a65fff7e2fa3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:02:41 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56446
I20260924 05:02:41.941017 17895 tablet_replica.cc:330] T a409478b3adc45d08c9a65fff7e2fa3e P d6b7e69b54b847869aa9dddfbd43a89d: stopping tablet replica
I20260924 05:02:41.941097 17895 raft_consensus.cc:2229] T a409478b3adc45d08c9a65fff7e2fa3e P d6b7e69b54b847869aa9dddfbd43a89d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:02:41.941187 17897 tablet_replica.cc:330] T a409478b3adc45d08c9a65fff7e2fa3e P f3aac06e9ab24aaba701f7f4ff7ea29d: stopping tablet replica
I20260924 05:02:41.941254 17895 raft_consensus.cc:2258] T a409478b3adc45d08c9a65fff7e2fa3e P d6b7e69b54b847869aa9dddfbd43a89d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:02:41.941334 17897 raft_consensus.cc:2229] T a409478b3adc45d08c9a65fff7e2fa3e P f3aac06e9ab24aaba701f7f4ff7ea29d [term 1 LEADER]: Raft consensus shutting down.
I20260924 05:02:41.941335 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 17461
I20260924 05:02:41.942111 17897 raft_consensus.cc:2258] T a409478b3adc45d08c9a65fff7e2fa3e P f3aac06e9ab24aaba701f7f4ff7ea29d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:02:41.942584 17897 ts_tablet_manager.cc:1905] T a409478b3adc45d08c9a65fff7e2fa3e P f3aac06e9ab24aaba701f7f4ff7ea29d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:02:41.944422 17897 ts_tablet_manager.cc:1918] T a409478b3adc45d08c9a65fff7e2fa3e P f3aac06e9ab24aaba701f7f4ff7ea29d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 05:02:41.944496 17897 log.cc:1196] T a409478b3adc45d08c9a65fff7e2fa3e P f3aac06e9ab24aaba701f7f4ff7ea29d: Deleting WAL directory at /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225929848825-392-0/minicluster-data/ts-1/wal/wals/a409478b3adc45d08c9a65fff7e2fa3e
I20260924 05:02:41.944810 17897 ts_tablet_manager.cc:1939] T a409478b3adc45d08c9a65fff7e2fa3e P f3aac06e9ab24aaba701f7f4ff7ea29d: Deleting consensus metadata
I20260924 05:02:41.945225 17386 catalog_manager.cc:4841] TS f3aac06e9ab24aaba701f7f4ff7ea29d (127.0.98.2:34927): tablet a409478b3adc45d08c9a65fff7e2fa3e (table test-table [id=d04f1447cbc54f91b7fe968f50f16af6]) successfully deleted
W20260924 05:02:41.946569 17385 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41509 (error 108)
I20260924 05:02:41.946729 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 17595
I20260924 05:02:41.946758 17896 tablet_replica.cc:330] T a409478b3adc45d08c9a65fff7e2fa3e P 0547c0fa87154077bd2ac8a970d2939e: stopping tablet replica
I20260924 05:02:41.946859 17896 raft_consensus.cc:2229] T a409478b3adc45d08c9a65fff7e2fa3e P 0547c0fa87154077bd2ac8a970d2939e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:02:41.946981 17896 raft_consensus.cc:2258] T a409478b3adc45d08c9a65fff7e2fa3e P 0547c0fa87154077bd2ac8a970d2939e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:02:41.947630 17896 ts_tablet_manager.cc:1905] T a409478b3adc45d08c9a65fff7e2fa3e P 0547c0fa87154077bd2ac8a970d2939e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:02:41.949203 17896 ts_tablet_manager.cc:1918] T a409478b3adc45d08c9a65fff7e2fa3e P 0547c0fa87154077bd2ac8a970d2939e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 05:02:41.949262 17896 log.cc:1196] T a409478b3adc45d08c9a65fff7e2fa3e P 0547c0fa87154077bd2ac8a970d2939e: Deleting WAL directory at /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790225929848825-392-0/minicluster-data/ts-2/wal/wals/a409478b3adc45d08c9a65fff7e2fa3e
I20260924 05:02:41.949563 17896 ts_tablet_manager.cc:1939] T a409478b3adc45d08c9a65fff7e2fa3e P 0547c0fa87154077bd2ac8a970d2939e: Deleting consensus metadata
I20260924 05:02:41.949852 17867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:41.949704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41509 (local address 127.0.0.1:47772)
0924 05:02:41.949766 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:41.949816 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260924 05:02:41.949968 17386 catalog_manager.cc:4841] TS 0547c0fa87154077bd2ac8a970d2939e (127.0.98.3:39275): tablet a409478b3adc45d08c9a65fff7e2fa3e (table test-table [id=d04f1447cbc54f91b7fe968f50f16af6]) successfully deleted
W20260924 05:02:41.949978 17385 catalog_manager.cc:4584] TS d6b7e69b54b847869aa9dddfbd43a89d (127.0.98.1:41509): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a409478b3adc45d08c9a65fff7e2fa3e: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41509: connect: Connection refused (error 111)
I20260924 05:02:41.952131 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 17729
I20260924 05:02:41.956311 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 17367
2026-09-24T05:02:41Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (1888 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-24T05:02:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:02:41Z Disabled control of system clock
I20260924 05:02:41.981235 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225929848825-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225929848825-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225929848825-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36463
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:43835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36463
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 05:02:42.058904 17902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:02:42.059119 17902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:02:42.059139 17902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:02:42.060335 17902 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:02:42.060379 17902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:02:42.060390 17902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:02:42.060401 17902 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:02:42.060411 17902 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:02:42.062038 17902 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225929848825-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225929848825-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36463
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36463
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17902
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225929848825-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:02:42.062278 17902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:02:42.062556 17902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:02:42.067076 17910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:02:42.067096 17908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:02:42.067263 17902 server_base.cc:1034] running on GCE node
W20260924 05:02:42.067093 17907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:02:42.067553 17902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:02:42.067840 17902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:02:42.069002 17902 hybrid_clock.cc:648] HybridClock initialized: now 1790226162068982 us; error 29 us; skew 500 ppm
I20260924 05:02:42.070272 17902 webserver.cc:460] Webserver started at http://127.0.98.62:32805/ using document root <none> and password file <none>
I20260924 05:02:42.070482 17902 fs_manager.cc:359] Metadata directory not provided
I20260924 05:02:42.070530 17902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:02:42.070650 17902 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:02:42.071688 17902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225929848825-392-0/minicluster-data/master-0/data/instance:
uuid: "a78e3ea41cbe460bbe9fd7e688e43397"
format_stamp: "Formatted at 2026-09-24 05:02:42 on dist-test-slave-70st"
server_key: "0d23af47b154260279ecdbf154b40ee8"
server_key_iv: "27b75806ec7ec657a8371ebc52c441ac"
server_key_version: "encryptionkey@0"
I20260924 05:02:42.072016 17902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225929848825-392-0/minicluster-data/master-0/wal/instance:
uuid: "a78e3ea41cbe460bbe9fd7e688e43397"
format_stamp: "Formatted at 2026-09-24 05:02:42 on dist-test-slave-70st"
server_key: "0d23af47b154260279ecdbf154b40ee8"
server_key_iv: "27b75806ec7ec657a8371ebc52c441ac"
server_key_version: "encryptionkey@0"
I20260924 05:02:42.073210 17902 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 05:02:42.073956 17916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:02:42.074117 17902 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 05:02:42.074199 17902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225929848825-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225929848825-392-0/minicluster-data/master-0/wal
uuid: "a78e3ea41cbe460bbe9fd7e688e43397"
format_stamp: "Formatted at 2026-09-24 05:02:42 on dist-test-slave-70st"
server_key: "0d23af47b154260279ecdbf154b40ee8"
server_key_iv: "27b75806ec7ec657a8371ebc52c441ac"
server_key_version: "encryptionkey@0"
I20260924 05:02:42.074265 17902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225929848825-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225929848825-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225929848825-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:02:42.096103 17902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:02:42.096405 17902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:02:42.096540 17902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:02:42.100158 17902 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36463
I20260924 05:02:42.100214 17968 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36463 every 8 connection(s)
I20260924 05:02:42.100521 17902 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:02:42.101450 17969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:02:42.103729 17969 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a78e3ea41cbe460bbe9fd7e688e43397: Bootstrap starting.
I20260924 05:02:42.104302 17969 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a78e3ea41cbe460bbe9fd7e688e43397: Neither blocks nor log segments found. Creating new log.
I20260924 05:02:42.104571 17969 log.cc:824] T 00000000000000000000000000000000 P a78e3ea41cbe460bbe9fd7e688e43397: Log is configured to *not* fsync() on all Append() calls
I20260924 05:02:42.105291 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 17902
I20260924 05:02:42.105399 17969 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a78e3ea41cbe460bbe9fd7e688e43397: No bootstrap required, opened a new log
I20260924 05:02:42.105437 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225929848825-392-0/minicluster-data/master-0/wal/instance
I20260924 05:02:42.105618 392 external_mini_cluster.cc:1428] Setting key 2709856d9b7e0c2853c6f1db7e9e24c2
I20260924 05:02:42.107313 17969 raft_consensus.cc:348] T 00000000000000000000000000000000 P a78e3ea41cbe460bbe9fd7e688e43397 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a78e3ea41cbe460bbe9fd7e688e43397" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36463 } }
I20260924 05:02:42.107471 17969 raft_consensus.cc:374] T 00000000000000000000000000000000 P a78e3ea41cbe460bbe9fd7e688e43397 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:02:42.107506 17969 raft_consensus.cc:729] T 00000000000000000000000000000000 P a78e3ea41cbe460bbe9fd7e688e43397 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a78e3ea41cbe460bbe9fd7e688e43397, State: Initialized, Role: FOLLOWER
I20260924 05:02:42.107666 17969 consensus_queue.cc:260] T 00000000000000000000000000000000 P a78e3ea41cbe460bbe9fd7e688e43397 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a78e3ea41cbe460bbe9fd7e688e43397" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36463 } }
I20260924 05:02:42.107750 17969 raft_consensus.cc:388] T 00000000000000000000000000000000 P a78e3ea41cbe460bbe9fd7e688e43397 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:02:42.107784 17969 raft_consensus.cc:482] T 00000000000000000000000000000000 P a78e3ea41cbe460bbe9fd7e688e43397 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:02:42.107831 17969 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a78e3ea41cbe460bbe9fd7e688e43397 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:02:42.108347 17969 raft_consensus.cc:504] T 00000000000000000000000000000000 P a78e3ea41cbe460bbe9fd7e688e43397 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a78e3ea41cbe460bbe9fd7e688e43397" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36463 } }
I20260924 05:02:42.108435 17969 leader_election.cc:302] T 00000000000000000000000000000000 P a78e3ea41cbe460bbe9fd7e688e43397 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a78e3ea41cbe460bbe9fd7e688e43397; no voters:
I20260924 05:02:42.108599 17969 leader_election.cc:288] T 00000000000000000000000000000000 P a78e3ea41cbe460bbe9fd7e688e43397 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:02:42.108639 17974 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a78e3ea41cbe460bbe9fd7e688e43397 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:02:42.108776 17974 raft_consensus.cc:686] T 00000000000000000000000000000000 P a78e3ea41cbe460bbe9fd7e688e43397 [term 1 LEADER]: Becoming Leader. State: Replica: a78e3ea41cbe460bbe9fd7e688e43397, State: Running, Role: LEADER
I20260924 05:02:42.108856 17969 sys_catalog.cc:564] T 00000000000000000000000000000000 P a78e3ea41cbe460bbe9fd7e688e43397 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:02:42.108911 17974 consensus_queue.cc:237] T 00000000000000000000000000000000 P a78e3ea41cbe460bbe9fd7e688e43397 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a78e3ea41cbe460bbe9fd7e688e43397" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36463 } }
I20260924 05:02:42.109297 17974 sys_catalog.cc:455] T 00000000000000000000000000000000 P a78e3ea41cbe460bbe9fd7e688e43397 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a78e3ea41cbe460bbe9fd7e688e43397. Latest consensus state: current_term: 1 leader_uuid: "a78e3ea41cbe460bbe9fd7e688e43397" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a78e3ea41cbe460bbe9fd7e688e43397" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36463 } } }
I20260924 05:02:42.109371 17974 sys_catalog.cc:458] T 00000000000000000000000000000000 P a78e3ea41cbe460bbe9fd7e688e43397 [sys.catalog]: This master's current role is: LEADER
I20260924 05:02:42.109543 17976 sys_catalog.cc:455] T 00000000000000000000000000000000 P a78e3ea41cbe460bbe9fd7e688e43397 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a78e3ea41cbe460bbe9fd7e688e43397" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a78e3ea41cbe460bbe9fd7e688e43397" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36463 } } }
I20260924 05:02:42.109602 17982 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:02:42.109680 17976 sys_catalog.cc:458] T 00000000000000000000000000000000 P a78e3ea41cbe460bbe9fd7e688e43397 [sys.catalog]: This master's current role is: LEADER
I20260924 05:02:42.110438 17982 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:02:42.112248 17973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:42.106198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32914 (local address 127.0.98.62:36463)
0924 05:02:42.106427 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0924 05:02:42.106435 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 05:02:42.106460 (+ 25us) server_negotiation.cc:407] Connection header received
0924 05:02:42.107276 (+ 816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:02:42.107282 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:02:42.107347 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:02:42.107478 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:02:42.107719 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:42.109186 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:42.110272 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:42.110845 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:42.110968 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:42.111649 (+ 681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:02:42.111652 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:02:42.111652 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:02:42.111802 (+ 150us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:02:42.111833 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:02:42.111843 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:02:42.111901 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 05:02:42.111988 (+ 87us) server_negotiation.cc:299] Negotiation successful
0924 05:02:42.112048 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":92,"threads_started":1}
I20260924 05:02:42.113922 17982 catalog_manager.cc:1347] Generated new cluster ID: 12f8438ee4774647a710874a3ab79ee5
I20260924 05:02:42.113981 17982 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:02:42.123356 17982 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:02:42.124054 17982 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:02:42.127853 17982 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a78e3ea41cbe460bbe9fd7e688e43397: Generated new TSK 0
I20260924 05:02:42.128036 17982 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 05:02:42.135211 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225929848825-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225929848825-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225929848825-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225929848825-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36463
--builtin_ntp_servers=127.0.98.20:43835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 05:02:42.215245 17993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:02:42.215463 17993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:02:42.215485 17993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:02:42.216642 17993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:02:42.216684 17993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:02:42.216709 17993 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 05:02:42.218106 17993 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225929848825-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225929848825-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225929848825-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36463
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17993
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225929848825-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:02:42.218343 17993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:02:42.218632 17993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:02:42.224257 18001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:02:42.224234 17998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:02:42.224424 17993 server_base.cc:1034] running on GCE node
W20260924 05:02:42.224282 17999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:02:42.224725 17993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:02:42.224989 17993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:02:42.226151 17993 hybrid_clock.cc:648] HybridClock initialized: now 1790226162226139 us; error 26 us; skew 500 ppm
I20260924 05:02:42.227309 17993 webserver.cc:460] Webserver started at http://127.0.98.1:45437/ using document root <none> and password file <none>
I20260924 05:02:42.227530 17993 fs_manager.cc:359] Metadata directory not provided
I20260924 05:02:42.227579 17993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:02:42.227696 17993 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:02:42.228747 17993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225929848825-392-0/minicluster-data/ts-0/data/instance:
uuid: "d1bb314d26b34f0e9942b9282188731b"
format_stamp: "Formatted at 2026-09-24 05:02:42 on dist-test-slave-70st"
server_key: "d352e34336befa697d8f28004aaef4cd"
server_key_iv: "087134d274d8bdbbf45ca78005c8c781"
server_key_version: "encryptionkey@0"
I20260924 05:02:42.229074 17993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225929848825-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d1bb314d26b34f0e9942b9282188731b"
format_stamp: "Formatted at 2026-09-24 05:02:42 on dist-test-slave-70st"
server_key: "d352e34336befa697d8f28004aaef4cd"
server_key_iv: "087134d274d8bdbbf45ca78005c8c781"
server_key_version: "encryptionkey@0"
I20260924 05:02:42.230360 17993 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 05:02:42.231055 18007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:02:42.231206 17993 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 05:02:42.231287 17993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225929848825-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225929848825-392-0/minicluster-data/ts-0/wal
uuid: "d1bb314d26b34f0e9942b9282188731b"
format_stamp: "Formatted at 2026-09-24 05:02:42 on dist-test-slave-70st"
server_key: "d352e34336befa697d8f28004aaef4cd"
server_key_iv: "087134d274d8bdbbf45ca78005c8c781"
server_key_version: "encryptionkey@0"
I20260924 05:02:42.231345 17993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225929848825-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225929848825-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225929848825-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:02:42.245075 17993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:02:42.245389 17993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:02:42.245553 17993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:02:42.245762 17993 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:02:42.246104 17993 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:02:42.246137 17993 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:02:42.246166 17993 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:02:42.246188 17993 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:02:42.251554 17993 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36283
I20260924 05:02:42.251628 18120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36283 every 8 connection(s)
I20260924 05:02:42.251931 17993 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225929848825-392-0/minicluster-data/ts-0/data/info.pb
I20260924 05:02:42.257077 17973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:42.252610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34921 (local address 127.0.98.62:36463)
0924 05:02:42.252690 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0924 05:02:42.252692 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:02:42.252900 (+ 208us) server_negotiation.cc:407] Connection header received
0924 05:02:42.253467 (+ 567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:02:42.253468 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:02:42.253484 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:02:42.253550 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:02:42.254226 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:42.254735 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:42.255734 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:42.255943 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:42.256002 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:42.256803 (+ 801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:02:42.256804 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:02:42.256804 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:02:42.256809 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:02:42.256824 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:02:42.256828 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:02:42.256880 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 05:02:42.256989 (+ 109us) server_negotiation.cc:299] Negotiation successful
0924 05:02:42.257016 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 05:02:42.257175 18123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:42.252587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36463 (local address 127.0.98.1:34921)
0924 05:02:42.252777 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:42.252812 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 05:02:42.253320 (+ 508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:02:42.253611 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:02:42.253614 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:02:42.253622 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:02:42.254122 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:42.254127 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:42.254817 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:42.254818 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:42.255640 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:42.255642 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:42.256044 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:42.256045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:42.256168 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:42.256671 (+ 503us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 05:02:42.256671 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:02:42.256699 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 05:02:42.256721 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:02:42.256883 (+ 162us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:02:42.256885 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:02:42.256887 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 05:02:42.256932 (+ 45us) client_negotiation.cc:240] Negotiation successful
0924 05:02:42.257011 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":76,"threads_started":1}
I20260924 05:02:42.257666 18121 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36463
I20260924 05:02:42.257748 18121 heartbeater.cc:464] Registering TS with master...
I20260924 05:02:42.257925 18121 heartbeater.cc:511] Master 127.0.98.62:36463 requested a full tablet report, sending...
I20260924 05:02:42.258352 17933 ts_manager.cc:194] Registered new tserver with Master: d1bb314d26b34f0e9942b9282188731b (127.0.98.1:36283)
I20260924 05:02:42.258980 17933 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:34921
I20260924 05:02:42.259725 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 17993
I20260924 05:02:42.259829 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225929848825-392-0/minicluster-data/ts-0/wal/instance
I20260924 05:02:42.260001 392 external_mini_cluster.cc:1428] Setting key f978c9691c94d04357a5022a6084dee7
I20260924 05:02:42.260870 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225929848825-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225929848825-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225929848825-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225929848825-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36463
--builtin_ntp_servers=127.0.98.20:43835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 05:02:42.340458 18124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:02:42.340657 18124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:02:42.340678 18124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:02:42.341840 18124 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:02:42.341879 18124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:02:42.341902 18124 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 05:02:42.343230 18124 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225929848825-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225929848825-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225929848825-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36463
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18124
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225929848825-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:02:42.343428 18124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:02:42.343667 18124 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:02:42.348313 18129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:02:42.348336 18130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:02:42.348357 18132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:02:42.348472 18124 server_base.cc:1034] running on GCE node
I20260924 05:02:42.348759 18124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:02:42.348974 18124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:02:42.350123 18124 hybrid_clock.cc:648] HybridClock initialized: now 1790226162350106 us; error 30 us; skew 500 ppm
I20260924 05:02:42.351328 18124 webserver.cc:460] Webserver started at http://127.0.98.2:38977/ using document root <none> and password file <none>
I20260924 05:02:42.351550 18124 fs_manager.cc:359] Metadata directory not provided
I20260924 05:02:42.351600 18124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:02:42.351724 18124 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:02:42.352739 18124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225929848825-392-0/minicluster-data/ts-1/data/instance:
uuid: "240c1d1eda844e9cb6b8d01ed328d0bf"
format_stamp: "Formatted at 2026-09-24 05:02:42 on dist-test-slave-70st"
server_key: "cbea512ebb6ed33d46e96a37058786c1"
server_key_iv: "a45a91b21841dbad1af05f87dd7cb9f5"
server_key_version: "encryptionkey@0"
I20260924 05:02:42.353067 18124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225929848825-392-0/minicluster-data/ts-1/wal/instance:
uuid: "240c1d1eda844e9cb6b8d01ed328d0bf"
format_stamp: "Formatted at 2026-09-24 05:02:42 on dist-test-slave-70st"
server_key: "cbea512ebb6ed33d46e96a37058786c1"
server_key_iv: "a45a91b21841dbad1af05f87dd7cb9f5"
server_key_version: "encryptionkey@0"
I20260924 05:02:42.354385 18124 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 05:02:42.355099 18138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:02:42.355230 18124 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 05:02:42.355312 18124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225929848825-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225929848825-392-0/minicluster-data/ts-1/wal
uuid: "240c1d1eda844e9cb6b8d01ed328d0bf"
format_stamp: "Formatted at 2026-09-24 05:02:42 on dist-test-slave-70st"
server_key: "cbea512ebb6ed33d46e96a37058786c1"
server_key_iv: "a45a91b21841dbad1af05f87dd7cb9f5"
server_key_version: "encryptionkey@0"
I20260924 05:02:42.355373 18124 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225929848825-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225929848825-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225929848825-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:02:42.366259 18124 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:02:42.366552 18124 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:02:42.366674 18124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:02:42.366887 18124 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:02:42.367182 18124 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:02:42.367215 18124 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:02:42.367245 18124 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:02:42.367278 18124 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:02:42.372611 18124 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39579
I20260924 05:02:42.372650 18251 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39579 every 8 connection(s)
I20260924 05:02:42.372961 18124 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225929848825-392-0/minicluster-data/ts-1/data/info.pb
I20260924 05:02:42.375190 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 18124
I20260924 05:02:42.375301 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225929848825-392-0/minicluster-data/ts-1/wal/instance
I20260924 05:02:42.375469 392 external_mini_cluster.cc:1428] Setting key e1c07b049144f9176cc3401d2fadaceb
I20260924 05:02:42.376420 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225929848825-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225929848825-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225929848825-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225929848825-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36463
--builtin_ntp_servers=127.0.98.20:43835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 05:02:42.378542 17973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:42.373678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43905 (local address 127.0.98.62:36463)
0924 05:02:42.373738 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 05:02:42.373740 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:02:42.373943 (+ 203us) server_negotiation.cc:407] Connection header received
0924 05:02:42.374421 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:02:42.374423 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:02:42.374435 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:02:42.374496 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:02:42.375163 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:42.375819 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:42.377124 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:42.377319 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:42.377381 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:42.378292 (+ 911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:02:42.378294 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:02:42.378294 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:02:42.378298 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:02:42.378315 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:02:42.378320 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:02:42.378377 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 05:02:42.378435 (+ 58us) server_negotiation.cc:299] Negotiation successful
0924 05:02:42.378464 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 05:02:42.378728 18254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:42.373654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36463 (local address 127.0.98.2:43905)
0924 05:02:42.373824 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:42.373864 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 05:02:42.374337 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:02:42.374550 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:02:42.374553 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:02:42.374560 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:02:42.375086 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:42.375093 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:42.375892 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:42.375893 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:42.377050 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:42.377053 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:42.377396 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:42.377397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:42.377612 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:42.378172 (+ 560us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 05:02:42.378172 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:02:42.378200 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 05:02:42.378222 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:02:42.378378 (+ 156us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:02:42.378381 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:02:42.378383 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 05:02:42.378423 (+ 40us) client_negotiation.cc:240] Negotiation successful
0924 05:02:42.378495 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":74,"threads_started":1}
I20260924 05:02:42.379168 18252 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36463
I20260924 05:02:42.379249 18252 heartbeater.cc:464] Registering TS with master...
I20260924 05:02:42.379426 18252 heartbeater.cc:511] Master 127.0.98.62:36463 requested a full tablet report, sending...
I20260924 05:02:42.379767 17933 ts_manager.cc:194] Registered new tserver with Master: 240c1d1eda844e9cb6b8d01ed328d0bf (127.0.98.2:39579)
I20260924 05:02:42.380123 17933 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:43905
W20260924 05:02:42.453855 18255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:02:42.454051 18255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:02:42.454069 18255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:02:42.455199 18255 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:02:42.455238 18255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:02:42.455261 18255 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 05:02:42.456553 18255 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225929848825-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225929848825-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225929848825-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36463
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18255
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225929848825-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:02:42.456766 18255 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:02:42.457036 18255 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:02:42.461807 18261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:02:42.461835 18255 server_base.cc:1034] running on GCE node
W20260924 05:02:42.461843 18260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:02:42.461808 18263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:02:42.462193 18255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:02:42.462401 18255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:02:42.463534 18255 hybrid_clock.cc:648] HybridClock initialized: now 1790226162463522 us; error 27 us; skew 500 ppm
I20260924 05:02:42.464741 18255 webserver.cc:460] Webserver started at http://127.0.98.3:35717/ using document root <none> and password file <none>
I20260924 05:02:42.464975 18255 fs_manager.cc:359] Metadata directory not provided
I20260924 05:02:42.465020 18255 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:02:42.465142 18255 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:02:42.466228 18255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225929848825-392-0/minicluster-data/ts-2/data/instance:
uuid: "538e5dda78ce4b68806552c9a61d2050"
format_stamp: "Formatted at 2026-09-24 05:02:42 on dist-test-slave-70st"
server_key: "80df89d2d83999fe6ed8131d451bbdcf"
server_key_iv: "45197aa8cdfc4fe3110ea7075d2055af"
server_key_version: "encryptionkey@0"
I20260924 05:02:42.466538 18255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225929848825-392-0/minicluster-data/ts-2/wal/instance:
uuid: "538e5dda78ce4b68806552c9a61d2050"
format_stamp: "Formatted at 2026-09-24 05:02:42 on dist-test-slave-70st"
server_key: "80df89d2d83999fe6ed8131d451bbdcf"
server_key_iv: "45197aa8cdfc4fe3110ea7075d2055af"
server_key_version: "encryptionkey@0"
I20260924 05:02:42.467739 18255 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 05:02:42.468387 18269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:02:42.468523 18255 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:02:42.468600 18255 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225929848825-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225929848825-392-0/minicluster-data/ts-2/wal
uuid: "538e5dda78ce4b68806552c9a61d2050"
format_stamp: "Formatted at 2026-09-24 05:02:42 on dist-test-slave-70st"
server_key: "80df89d2d83999fe6ed8131d451bbdcf"
server_key_iv: "45197aa8cdfc4fe3110ea7075d2055af"
server_key_version: "encryptionkey@0"
I20260924 05:02:42.468658 18255 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225929848825-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225929848825-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225929848825-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:02:42.485383 18255 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:02:42.485767 18255 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:02:42.485894 18255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:02:42.486109 18255 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:02:42.486420 18255 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:02:42.486455 18255 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:02:42.486476 18255 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:02:42.486495 18255 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:02:42.491619 18255 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34215
I20260924 05:02:42.491706 18382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34215 every 8 connection(s)
I20260924 05:02:42.491976 18255 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225929848825-392-0/minicluster-data/ts-2/data/info.pb
I20260924 05:02:42.497299 17973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:42.492753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33251 (local address 127.0.98.62:36463)
0924 05:02:42.492843 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0924 05:02:42.492845 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:02:42.493014 (+ 169us) server_negotiation.cc:407] Connection header received
0924 05:02:42.493712 (+ 698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:02:42.493714 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:02:42.493727 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:02:42.493791 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:02:42.494464 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:42.494963 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:42.495960 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:42.496160 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:42.496215 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:42.497039 (+ 824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:02:42.497041 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:02:42.497041 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:02:42.497046 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:02:42.497063 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:02:42.497066 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:02:42.497111 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 05:02:42.497184 (+ 73us) server_negotiation.cc:299] Negotiation successful
0924 05:02:42.497215 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 05:02:42.497524 18385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:42.492670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36463 (local address 127.0.98.3:33251)
0924 05:02:42.492877 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:42.492921 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 05:02:42.493601 (+ 680us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:02:42.493866 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:02:42.493870 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:02:42.493881 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:02:42.494373 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:42.494378 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:42.495059 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:42.495060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:42.495871 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:42.495873 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:42.496232 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:42.496233 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:42.496374 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:42.496904 (+ 530us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 05:02:42.496905 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:02:42.496934 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 05:02:42.496955 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:02:42.497116 (+ 161us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:02:42.497120 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:02:42.497124 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 05:02:42.497187 (+ 63us) client_negotiation.cc:240] Negotiation successful
0924 05:02:42.497292 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":82,"threads_started":1}
I20260924 05:02:42.497964 18383 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36463
I20260924 05:02:42.498047 18383 heartbeater.cc:464] Registering TS with master...
I20260924 05:02:42.498211 18383 heartbeater.cc:511] Master 127.0.98.62:36463 requested a full tablet report, sending...
I20260924 05:02:42.498569 17933 ts_manager.cc:194] Registered new tserver with Master: 538e5dda78ce4b68806552c9a61d2050 (127.0.98.3:34215)
I20260924 05:02:42.498919 17933 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:33251
I20260924 05:02:42.501392 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 18255
I20260924 05:02:42.501508 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225929848825-392-0/minicluster-data/ts-2/wal/instance
I20260924 05:02:42.501659 392 external_mini_cluster.cc:1428] Setting key aaf5a3f8f213b3d444f239376f3197e5
I20260924 05:02:42.502378 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 05:02:42.506249 17973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:42.503615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32916 (local address 127.0.98.62:36463)
0924 05:02:42.503663 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 05:02:42.503665 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:02:42.503768 (+ 103us) server_negotiation.cc:407] Connection header received
0924 05:02:42.503825 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:02:42.503826 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:02:42.503835 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:02:42.503871 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:02:42.503994 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:42.504448 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:42.505194 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:42.505371 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:42.505411 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:42.506086 (+ 675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:02:42.506087 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:02:42.506087 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:02:42.506090 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:02:42.506100 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:02:42.506102 (+ 2us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:02:42.506136 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0924 05:02:42.506179 (+ 43us) server_negotiation.cc:299] Negotiation successful
0924 05:02:42.506197 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 05:02:42.507683 17933 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:32916:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 05:02:42.509276 17933 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 05:02:42.515286 18396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:42.511537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34312 (local address 127.0.98.3:34215)
0924 05:02:42.511703 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 05:02:42.511707 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 05:02:42.511717 (+ 10us) server_negotiation.cc:407] Connection header received
0924 05:02:42.511813 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:02:42.511817 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:02:42.511832 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:02:42.511899 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:02:42.512129 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:42.513161 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:42.514224 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:42.514889 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:42.514961 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:42.515066 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 05:02:42.515162 (+ 96us) server_negotiation.cc:299] Negotiation successful
0924 05:02:42.515200 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":75,"threads_started":1}
I20260924 05:02:42.515283 18394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:42.511190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34215 (local address 127.0.0.1:34312)
0924 05:02:42.511662 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:42.511671 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:02:42.511717 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:02:42.511963 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:02:42.511964 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:02:42.511966 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:02:42.512059 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:42.512061 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:42.513216 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:42.513217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:42.514155 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:42.514158 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:42.514991 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:42.514992 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:42.515068 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:42.515082 (+ 14us) client_negotiation.cc:769] Sending connection context
0924 05:02:42.515149 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 05:02:42.515198 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"mutex_wait_us":2,"thread_start_us":72,"threads_started":1}
I20260924 05:02:42.515414 18392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:42.511190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36283 (local address 127.0.0.1:53836)
0924 05:02:42.511329 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:42.511370 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 05:02:42.511436 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:02:42.511731 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:02:42.511734 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:02:42.511736 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:02:42.511806 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:42.511808 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:42.513148 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:42.513149 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:42.514336 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:42.514339 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:42.515198 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:42.515199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:42.515259 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:42.515263 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 05:02:42.515292 (+ 29us) client_negotiation.cc:240] Negotiation successful
0924 05:02:42.515341 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"mutex_wait_us":5,"thread_start_us":77,"threads_started":1}
I20260924 05:02:42.515444 18395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:42.511503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54624 (local address 127.0.98.2:39579)
0924 05:02:42.511655 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 05:02:42.511660 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 05:02:42.511818 (+ 158us) server_negotiation.cc:407] Connection header received
0924 05:02:42.511889 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:02:42.511892 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:02:42.511906 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:02:42.511971 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:02:42.512147 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:42.512915 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:42.514431 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:42.514908 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:42.514981 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:42.515070 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 05:02:42.515320 (+ 250us) server_negotiation.cc:299] Negotiation successful
0924 05:02:42.515362 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":76,"threads_started":1}
I20260924 05:02:42.515565 18391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:42.511400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39579 (local address 127.0.0.1:54624)
0924 05:02:42.511762 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:42.511772 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:02:42.511819 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:02:42.512033 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:02:42.512034 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:02:42.512036 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:02:42.512077 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:42.512079 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:42.512994 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:42.512995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:42.514336 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:42.514339 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:42.515100 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:42.515101 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:42.515172 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:42.515175 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:02:42.515215 (+ 40us) client_negotiation.cc:240] Negotiation successful
0924 05:02:42.515255 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"mutex_wait_us":78,"thread_start_us":74,"threads_started":1}
I20260924 05:02:42.515700 18393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:42.511277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53836 (local address 127.0.98.1:36283)
0924 05:02:42.511564 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0924 05:02:42.511574 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 05:02:42.511586 (+ 12us) server_negotiation.cc:407] Connection header received
0924 05:02:42.511627 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:02:42.511629 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:02:42.511644 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:02:42.511728 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:02:42.511909 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:42.513062 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:42.514416 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:42.515110 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:42.515470 (+ 360us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:42.515565 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 05:02:42.515588 (+ 23us) server_negotiation.cc:299] Negotiation successful
0924 05:02:42.515624 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":96,"threads_started":1}
I20260924 05:02:42.516259 18186 tablet_service.cc:1467] Processing CreateTablet for tablet 82e2be3ef11a40ccbf128cddfb49a714 (DEFAULT_TABLE table=test-table [id=a48df781da244385b9a3d025e8ce7e5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:02:42.516335 18055 tablet_service.cc:1467] Processing CreateTablet for tablet 82e2be3ef11a40ccbf128cddfb49a714 (DEFAULT_TABLE table=test-table [id=a48df781da244385b9a3d025e8ce7e5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:02:42.516667 18055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82e2be3ef11a40ccbf128cddfb49a714. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:02:42.516669 18186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82e2be3ef11a40ccbf128cddfb49a714. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:02:42.519428 18397 tablet_bootstrap.cc:492] T 82e2be3ef11a40ccbf128cddfb49a714 P 240c1d1eda844e9cb6b8d01ed328d0bf: Bootstrap starting.
I20260924 05:02:42.519614 18398 tablet_bootstrap.cc:492] T 82e2be3ef11a40ccbf128cddfb49a714 P d1bb314d26b34f0e9942b9282188731b: Bootstrap starting.
I20260924 05:02:42.520057 18397 tablet_bootstrap.cc:654] T 82e2be3ef11a40ccbf128cddfb49a714 P 240c1d1eda844e9cb6b8d01ed328d0bf: Neither blocks nor log segments found. Creating new log.
I20260924 05:02:42.520231 18398 tablet_bootstrap.cc:654] T 82e2be3ef11a40ccbf128cddfb49a714 P d1bb314d26b34f0e9942b9282188731b: Neither blocks nor log segments found. Creating new log.
I20260924 05:02:42.520179 18317 tablet_service.cc:1467] Processing CreateTablet for tablet 82e2be3ef11a40ccbf128cddfb49a714 (DEFAULT_TABLE table=test-table [id=a48df781da244385b9a3d025e8ce7e5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:02:42.520360 18397 log.cc:824] T 82e2be3ef11a40ccbf128cddfb49a714 P 240c1d1eda844e9cb6b8d01ed328d0bf: Log is configured to *not* fsync() on all Append() calls
I20260924 05:02:42.520511 18398 log.cc:824] T 82e2be3ef11a40ccbf128cddfb49a714 P d1bb314d26b34f0e9942b9282188731b: Log is configured to *not* fsync() on all Append() calls
I20260924 05:02:42.520529 18317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82e2be3ef11a40ccbf128cddfb49a714. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:02:42.521147 18397 tablet_bootstrap.cc:492] T 82e2be3ef11a40ccbf128cddfb49a714 P 240c1d1eda844e9cb6b8d01ed328d0bf: No bootstrap required, opened a new log
I20260924 05:02:42.521188 18398 tablet_bootstrap.cc:492] T 82e2be3ef11a40ccbf128cddfb49a714 P d1bb314d26b34f0e9942b9282188731b: No bootstrap required, opened a new log
I20260924 05:02:42.521224 18397 ts_tablet_manager.cc:1397] T 82e2be3ef11a40ccbf128cddfb49a714 P 240c1d1eda844e9cb6b8d01ed328d0bf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 05:02:42.521245 18398 ts_tablet_manager.cc:1397] T 82e2be3ef11a40ccbf128cddfb49a714 P d1bb314d26b34f0e9942b9282188731b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 05:02:42.523068 18401 tablet_bootstrap.cc:492] T 82e2be3ef11a40ccbf128cddfb49a714 P 538e5dda78ce4b68806552c9a61d2050: Bootstrap starting.
I20260924 05:02:42.523626 18401 tablet_bootstrap.cc:654] T 82e2be3ef11a40ccbf128cddfb49a714 P 538e5dda78ce4b68806552c9a61d2050: Neither blocks nor log segments found. Creating new log.
I20260924 05:02:42.523670 18398 raft_consensus.cc:348] T 82e2be3ef11a40ccbf128cddfb49a714 P d1bb314d26b34f0e9942b9282188731b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1bb314d26b34f0e9942b9282188731b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36283 } } peers { permanent_uuid: "538e5dda78ce4b68806552c9a61d2050" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34215 } } peers { permanent_uuid: "240c1d1eda844e9cb6b8d01ed328d0bf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39579 } }
I20260924 05:02:42.523634 18397 raft_consensus.cc:348] T 82e2be3ef11a40ccbf128cddfb49a714 P 240c1d1eda844e9cb6b8d01ed328d0bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1bb314d26b34f0e9942b9282188731b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36283 } } peers { permanent_uuid: "538e5dda78ce4b68806552c9a61d2050" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34215 } } peers { permanent_uuid: "240c1d1eda844e9cb6b8d01ed328d0bf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39579 } }
I20260924 05:02:42.523813 18398 raft_consensus.cc:374] T 82e2be3ef11a40ccbf128cddfb49a714 P d1bb314d26b34f0e9942b9282188731b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:02:42.523813 18397 raft_consensus.cc:374] T 82e2be3ef11a40ccbf128cddfb49a714 P 240c1d1eda844e9cb6b8d01ed328d0bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:02:42.523852 18397 raft_consensus.cc:729] T 82e2be3ef11a40ccbf128cddfb49a714 P 240c1d1eda844e9cb6b8d01ed328d0bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 240c1d1eda844e9cb6b8d01ed328d0bf, State: Initialized, Role: FOLLOWER
I20260924 05:02:42.523852 18398 raft_consensus.cc:729] T 82e2be3ef11a40ccbf128cddfb49a714 P d1bb314d26b34f0e9942b9282188731b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1bb314d26b34f0e9942b9282188731b, State: Initialized, Role: FOLLOWER
I20260924 05:02:42.523914 18401 log.cc:824] T 82e2be3ef11a40ccbf128cddfb49a714 P 538e5dda78ce4b68806552c9a61d2050: Log is configured to *not* fsync() on all Append() calls
I20260924 05:02:42.523983 18397 consensus_queue.cc:260] T 82e2be3ef11a40ccbf128cddfb49a714 P 240c1d1eda844e9cb6b8d01ed328d0bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1bb314d26b34f0e9942b9282188731b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36283 } } peers { permanent_uuid: "538e5dda78ce4b68806552c9a61d2050" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34215 } } peers { permanent_uuid: "240c1d1eda844e9cb6b8d01ed328d0bf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39579 } }
I20260924 05:02:42.523983 18398 consensus_queue.cc:260] T 82e2be3ef11a40ccbf128cddfb49a714 P d1bb314d26b34f0e9942b9282188731b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1bb314d26b34f0e9942b9282188731b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36283 } } peers { permanent_uuid: "538e5dda78ce4b68806552c9a61d2050" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34215 } } peers { permanent_uuid: "240c1d1eda844e9cb6b8d01ed328d0bf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39579 } }
I20260924 05:02:42.524235 18397 ts_tablet_manager.cc:1428] T 82e2be3ef11a40ccbf128cddfb49a714 P 240c1d1eda844e9cb6b8d01ed328d0bf: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 05:02:42.524245 18398 ts_tablet_manager.cc:1428] T 82e2be3ef11a40ccbf128cddfb49a714 P d1bb314d26b34f0e9942b9282188731b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 05:02:42.524361 18252 heartbeater.cc:503] Master 127.0.98.62:36463 was elected leader, sending a full tablet report...
I20260924 05:02:42.524569 18401 tablet_bootstrap.cc:492] T 82e2be3ef11a40ccbf128cddfb49a714 P 538e5dda78ce4b68806552c9a61d2050: No bootstrap required, opened a new log
I20260924 05:02:42.524610 18121 heartbeater.cc:503] Master 127.0.98.62:36463 was elected leader, sending a full tablet report...
I20260924 05:02:42.524622 18401 ts_tablet_manager.cc:1397] T 82e2be3ef11a40ccbf128cddfb49a714 P 538e5dda78ce4b68806552c9a61d2050: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 05:02:42.526525 18401 raft_consensus.cc:348] T 82e2be3ef11a40ccbf128cddfb49a714 P 538e5dda78ce4b68806552c9a61d2050 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1bb314d26b34f0e9942b9282188731b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36283 } } peers { permanent_uuid: "538e5dda78ce4b68806552c9a61d2050" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34215 } } peers { permanent_uuid: "240c1d1eda844e9cb6b8d01ed328d0bf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39579 } }
I20260924 05:02:42.526661 18401 raft_consensus.cc:374] T 82e2be3ef11a40ccbf128cddfb49a714 P 538e5dda78ce4b68806552c9a61d2050 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:02:42.526705 18401 raft_consensus.cc:729] T 82e2be3ef11a40ccbf128cddfb49a714 P 538e5dda78ce4b68806552c9a61d2050 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 538e5dda78ce4b68806552c9a61d2050, State: Initialized, Role: FOLLOWER
I20260924 05:02:42.526809 18401 consensus_queue.cc:260] T 82e2be3ef11a40ccbf128cddfb49a714 P 538e5dda78ce4b68806552c9a61d2050 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1bb314d26b34f0e9942b9282188731b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36283 } } peers { permanent_uuid: "538e5dda78ce4b68806552c9a61d2050" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34215 } } peers { permanent_uuid: "240c1d1eda844e9cb6b8d01ed328d0bf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39579 } }
I20260924 05:02:42.527033 18401 ts_tablet_manager.cc:1428] T 82e2be3ef11a40ccbf128cddfb49a714 P 538e5dda78ce4b68806552c9a61d2050: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 05:02:42.527086 18383 heartbeater.cc:503] Master 127.0.98.62:36463 was elected leader, sending a full tablet report...
W20260924 05:02:42.623504 18253 tablet.cc:2369] T 82e2be3ef11a40ccbf128cddfb49a714 P 240c1d1eda844e9cb6b8d01ed328d0bf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 05:02:42.742515 18384 tablet.cc:2369] T 82e2be3ef11a40ccbf128cddfb49a714 P 538e5dda78ce4b68806552c9a61d2050: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 05:02:42.752617 18122 tablet.cc:2369] T 82e2be3ef11a40ccbf128cddfb49a714 P d1bb314d26b34f0e9942b9282188731b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 05:02:42.782888 18404 raft_consensus.cc:482] T 82e2be3ef11a40ccbf128cddfb49a714 P 240c1d1eda844e9cb6b8d01ed328d0bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 05:02:42.783016 18404 raft_consensus.cc:504] T 82e2be3ef11a40ccbf128cddfb49a714 P 240c1d1eda844e9cb6b8d01ed328d0bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1bb314d26b34f0e9942b9282188731b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36283 } } peers { permanent_uuid: "538e5dda78ce4b68806552c9a61d2050" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34215 } } peers { permanent_uuid: "240c1d1eda844e9cb6b8d01ed328d0bf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39579 } }
I20260924 05:02:42.783375 18404 leader_election.cc:288] T 82e2be3ef11a40ccbf128cddfb49a714 P 240c1d1eda844e9cb6b8d01ed328d0bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1bb314d26b34f0e9942b9282188731b (127.0.98.1:36283), 538e5dda78ce4b68806552c9a61d2050 (127.0.98.3:34215)
I20260924 05:02:42.786201 18254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:42.783569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34215 (local address 127.0.98.2:51903)
0924 05:02:42.783610 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:42.783622 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 05:02:42.783667 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:02:42.783817 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:02:42.783818 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:02:42.783819 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:02:42.783886 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:42.783888 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:42.784602 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:42.784603 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:42.785407 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:42.785409 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:42.785977 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:42.785978 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:42.786045 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:42.786048 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:02:42.786092 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 05:02:42.786125 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21,"mutex_wait_us":43}
I20260924 05:02:42.786227 18396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:42.783637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51903 (local address 127.0.98.3:34215)
0924 05:02:42.783690 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 05:02:42.783693 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:02:42.783705 (+ 12us) server_negotiation.cc:407] Connection header received
0924 05:02:42.783733 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:02:42.783734 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:02:42.783742 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:02:42.783797 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:02:42.784066 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:42.784524 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:42.785514 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:42.785902 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:42.785954 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:42.785963 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 05:02:42.786113 (+ 150us) server_negotiation.cc:299] Negotiation successful
0924 05:02:42.786157 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 05:02:42.786260 18393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:42.783657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51125 (local address 127.0.98.1:36283)
0924 05:02:42.783703 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 05:02:42.783705 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:02:42.783956 (+ 251us) server_negotiation.cc:407] Connection header received
0924 05:02:42.783982 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:02:42.783983 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:02:42.783992 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:02:42.784028 (+ 36us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:02:42.784173 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:42.784633 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:42.785585 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:42.785946 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:42.786013 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:42.786022 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 05:02:42.786168 (+ 146us) server_negotiation.cc:299] Negotiation successful
0924 05:02:42.786202 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 05:02:42.786497 18406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:42.783575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36283 (local address 127.0.98.2:51125)
0924 05:02:42.783871 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:42.783878 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0924 05:02:42.783927 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:02:42.784039 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:02:42.784040 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:02:42.784041 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:02:42.784097 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:42.784099 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:42.784710 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:42.784711 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:42.785505 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:42.785507 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:42.786028 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:42.786029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:42.786095 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:42.786100 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 05:02:42.786151 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 05:02:42.786185 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"mutex_wait_us":140,"thread_start_us":96,"threads_started":1}
I20260924 05:02:42.786592 18337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82e2be3ef11a40ccbf128cddfb49a714" candidate_uuid: "240c1d1eda844e9cb6b8d01ed328d0bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "538e5dda78ce4b68806552c9a61d2050" is_pre_election: true
I20260924 05:02:42.786758 18337 raft_consensus.cc:2454] T 82e2be3ef11a40ccbf128cddfb49a714 P 538e5dda78ce4b68806552c9a61d2050 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 240c1d1eda844e9cb6b8d01ed328d0bf in term 0.
I20260924 05:02:42.786790 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82e2be3ef11a40ccbf128cddfb49a714" candidate_uuid: "240c1d1eda844e9cb6b8d01ed328d0bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1bb314d26b34f0e9942b9282188731b" is_pre_election: true
I20260924 05:02:42.786922 18075 raft_consensus.cc:2454] T 82e2be3ef11a40ccbf128cddfb49a714 P d1bb314d26b34f0e9942b9282188731b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 240c1d1eda844e9cb6b8d01ed328d0bf in term 0.
I20260924 05:02:42.787007 18139 leader_election.cc:302] T 82e2be3ef11a40ccbf128cddfb49a714 P 240c1d1eda844e9cb6b8d01ed328d0bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 240c1d1eda844e9cb6b8d01ed328d0bf, 538e5dda78ce4b68806552c9a61d2050; no voters:
I20260924 05:02:42.787122 18404 raft_consensus.cc:2781] T 82e2be3ef11a40ccbf128cddfb49a714 P 240c1d1eda844e9cb6b8d01ed328d0bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 05:02:42.787170 18404 raft_consensus.cc:482] T 82e2be3ef11a40ccbf128cddfb49a714 P 240c1d1eda844e9cb6b8d01ed328d0bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 05:02:42.787192 18404 raft_consensus.cc:3037] T 82e2be3ef11a40ccbf128cddfb49a714 P 240c1d1eda844e9cb6b8d01ed328d0bf [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:02:42.787849 18404 raft_consensus.cc:504] T 82e2be3ef11a40ccbf128cddfb49a714 P 240c1d1eda844e9cb6b8d01ed328d0bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1bb314d26b34f0e9942b9282188731b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36283 } } peers { permanent_uuid: "538e5dda78ce4b68806552c9a61d2050" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34215 } } peers { permanent_uuid: "240c1d1eda844e9cb6b8d01ed328d0bf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39579 } }
I20260924 05:02:42.787969 18404 leader_election.cc:288] T 82e2be3ef11a40ccbf128cddfb49a714 P 240c1d1eda844e9cb6b8d01ed328d0bf [CANDIDATE]: Term 1 election: Requested vote from peers d1bb314d26b34f0e9942b9282188731b (127.0.98.1:36283), 538e5dda78ce4b68806552c9a61d2050 (127.0.98.3:34215)
I20260924 05:02:42.788113 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82e2be3ef11a40ccbf128cddfb49a714" candidate_uuid: "240c1d1eda844e9cb6b8d01ed328d0bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1bb314d26b34f0e9942b9282188731b"
I20260924 05:02:42.788111 18337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82e2be3ef11a40ccbf128cddfb49a714" candidate_uuid: "240c1d1eda844e9cb6b8d01ed328d0bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "538e5dda78ce4b68806552c9a61d2050"
I20260924 05:02:42.788177 18075 raft_consensus.cc:3037] T 82e2be3ef11a40ccbf128cddfb49a714 P d1bb314d26b34f0e9942b9282188731b [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:02:42.788182 18337 raft_consensus.cc:3037] T 82e2be3ef11a40ccbf128cddfb49a714 P 538e5dda78ce4b68806552c9a61d2050 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:02:42.788844 18075 raft_consensus.cc:2454] T 82e2be3ef11a40ccbf128cddfb49a714 P d1bb314d26b34f0e9942b9282188731b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 240c1d1eda844e9cb6b8d01ed328d0bf in term 1.
I20260924 05:02:42.788888 18337 raft_consensus.cc:2454] T 82e2be3ef11a40ccbf128cddfb49a714 P 538e5dda78ce4b68806552c9a61d2050 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 240c1d1eda844e9cb6b8d01ed328d0bf in term 1.
I20260924 05:02:42.789017 18141 leader_election.cc:302] T 82e2be3ef11a40ccbf128cddfb49a714 P 240c1d1eda844e9cb6b8d01ed328d0bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 240c1d1eda844e9cb6b8d01ed328d0bf, d1bb314d26b34f0e9942b9282188731b; no voters:
I20260924 05:02:42.789150 18404 raft_consensus.cc:2781] T 82e2be3ef11a40ccbf128cddfb49a714 P 240c1d1eda844e9cb6b8d01ed328d0bf [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:02:42.789280 18404 raft_consensus.cc:686] T 82e2be3ef11a40ccbf128cddfb49a714 P 240c1d1eda844e9cb6b8d01ed328d0bf [term 1 LEADER]: Becoming Leader. State: Replica: 240c1d1eda844e9cb6b8d01ed328d0bf, State: Running, Role: LEADER
I20260924 05:02:42.789366 18404 consensus_queue.cc:237] T 82e2be3ef11a40ccbf128cddfb49a714 P 240c1d1eda844e9cb6b8d01ed328d0bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1bb314d26b34f0e9942b9282188731b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36283 } } peers { permanent_uuid: "538e5dda78ce4b68806552c9a61d2050" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34215 } } peers { permanent_uuid: "240c1d1eda844e9cb6b8d01ed328d0bf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39579 } }
I20260924 05:02:42.790155 17932 catalog_manager.cc:5495] T 82e2be3ef11a40ccbf128cddfb49a714 P 240c1d1eda844e9cb6b8d01ed328d0bf reported cstate change: term changed from 0 to 1, leader changed from <none> to 240c1d1eda844e9cb6b8d01ed328d0bf (127.0.98.2). New cstate: current_term: 1 leader_uuid: "240c1d1eda844e9cb6b8d01ed328d0bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1bb314d26b34f0e9942b9282188731b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "538e5dda78ce4b68806552c9a61d2050" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "240c1d1eda844e9cb6b8d01ed328d0bf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39579 } health_report { overall_health: HEALTHY } } }
I20260924 05:02:42.795753 18395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:42.793652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54632 (local address 127.0.98.2:39579)
0924 05:02:42.793694 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 05:02:42.793696 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:02:42.793729 (+ 33us) server_negotiation.cc:407] Connection header received
0924 05:02:42.793788 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:02:42.793788 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:02:42.793802 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:02:42.793861 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:02:42.793952 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:42.794420 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:42.795140 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:42.795313 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:42.795377 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:42.795492 (+ 115us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:02:42.795581 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:02:42.795631 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 05:02:42.795666 (+ 35us) server_negotiation.cc:299] Negotiation successful
0924 05:02:42.795678 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 05:02:42.797371 18337 raft_consensus.cc:1261] T 82e2be3ef11a40ccbf128cddfb49a714 P 538e5dda78ce4b68806552c9a61d2050 [term 1 FOLLOWER]: Refusing update from remote peer 240c1d1eda844e9cb6b8d01ed328d0bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:02:42.797405 18075 raft_consensus.cc:1261] T 82e2be3ef11a40ccbf128cddfb49a714 P d1bb314d26b34f0e9942b9282188731b [term 1 FOLLOWER]: Refusing update from remote peer 240c1d1eda844e9cb6b8d01ed328d0bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:02:42.797736 18407 consensus_queue.cc:1035] T 82e2be3ef11a40ccbf128cddfb49a714 P 240c1d1eda844e9cb6b8d01ed328d0bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "538e5dda78ce4b68806552c9a61d2050" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:02:42.797852 18404 consensus_queue.cc:1035] T 82e2be3ef11a40ccbf128cddfb49a714 P 240c1d1eda844e9cb6b8d01ed328d0bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1bb314d26b34f0e9942b9282188731b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:02:42.800027 18410 mvcc.cc:205] Tried to move back new op lower bound from 7332766362815651840 to 7332766362785849344. Current Snapshot: MvccSnapshot[applied={T|T < 7332766362815651840}]
I20260924 05:02:42.800221 18412 mvcc.cc:205] Tried to move back new op lower bound from 7332766362815651840 to 7332766362785849344. Current Snapshot: MvccSnapshot[applied={T|T < 7332766362815651840}]
I20260924 05:02:42.800719 18411 mvcc.cc:205] Tried to move back new op lower bound from 7332766362815651840 to 7332766362785849344. Current Snapshot: MvccSnapshot[applied={T|T < 7332766362815651840}]
I20260924 05:02:42.802073 17932 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:32916:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:02:42.802156 17932 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:32916:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:02:42.802846 17932 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a78e3ea41cbe460bbe9fd7e688e43397: Sending DeleteTablet for 3 replicas of tablet 82e2be3ef11a40ccbf128cddfb49a714
I20260924 05:02:42.803126 18055 tablet_service.cc:1514] Processing DeleteTablet for tablet 82e2be3ef11a40ccbf128cddfb49a714 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:02:42 UTC) from {username='slave'} at 127.0.0.1:53836
I20260924 05:02:42.803202 18186 tablet_service.cc:1514] Processing DeleteTablet for tablet 82e2be3ef11a40ccbf128cddfb49a714 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:02:42 UTC) from {username='slave'} at 127.0.0.1:54624
I20260924 05:02:42.803223 18317 tablet_service.cc:1514] Processing DeleteTablet for tablet 82e2be3ef11a40ccbf128cddfb49a714 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:02:42 UTC) from {username='slave'} at 127.0.0.1:34312
I20260924 05:02:42.803392 18421 tablet_replica.cc:330] T 82e2be3ef11a40ccbf128cddfb49a714 P 240c1d1eda844e9cb6b8d01ed328d0bf: stopping tablet replica
I20260924 05:02:42.803498 18421 raft_consensus.cc:2229] T 82e2be3ef11a40ccbf128cddfb49a714 P 240c1d1eda844e9cb6b8d01ed328d0bf [term 1 LEADER]: Raft consensus shutting down.
I20260924 05:02:42.803639 18422 tablet_replica.cc:330] T 82e2be3ef11a40ccbf128cddfb49a714 P 538e5dda78ce4b68806552c9a61d2050: stopping tablet replica
I20260924 05:02:42.803714 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 17993
I20260924 05:02:42.803728 18422 raft_consensus.cc:2229] T 82e2be3ef11a40ccbf128cddfb49a714 P 538e5dda78ce4b68806552c9a61d2050 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:02:42.803817 18422 raft_consensus.cc:2258] T 82e2be3ef11a40ccbf128cddfb49a714 P 538e5dda78ce4b68806552c9a61d2050 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:02:42.804790 18422 ts_tablet_manager.cc:1905] T 82e2be3ef11a40ccbf128cddfb49a714 P 538e5dda78ce4b68806552c9a61d2050: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:02:42.806504 18422 ts_tablet_manager.cc:1918] T 82e2be3ef11a40ccbf128cddfb49a714 P 538e5dda78ce4b68806552c9a61d2050: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 05:02:42.806576 18422 log.cc:1196] T 82e2be3ef11a40ccbf128cddfb49a714 P 538e5dda78ce4b68806552c9a61d2050: Deleting WAL directory at /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790225929848825-392-0/minicluster-data/ts-2/wal/wals/82e2be3ef11a40ccbf128cddfb49a714
I20260924 05:02:42.806802 18422 ts_tablet_manager.cc:1939] T 82e2be3ef11a40ccbf128cddfb49a714 P 538e5dda78ce4b68806552c9a61d2050: Deleting consensus metadata
I20260924 05:02:42.807216 17917 catalog_manager.cc:4841] TS 538e5dda78ce4b68806552c9a61d2050 (127.0.98.3:34215): tablet 82e2be3ef11a40ccbf128cddfb49a714 (table test-table [id=a48df781da244385b9a3d025e8ce7e5e]) successfully deleted
W20260924 05:02:42.809103 17919 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36283 (error 108)
I20260924 05:02:42.809288 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 18124
I20260924 05:02:42.810698 18394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:42.810544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36283 (local address 127.0.0.1:53838)
0924 05:02:42.810621 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:42.810668 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260924 05:02:42.810860 17919 catalog_manager.cc:4584] TS d1bb314d26b34f0e9942b9282188731b (127.0.98.1:36283): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 82e2be3ef11a40ccbf128cddfb49a714: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36283: connect: Connection refused (error 111)
I20260924 05:02:42.813663 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 18255
I20260924 05:02:42.813725 18394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:42.813631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39579 (local address 127.0.0.1:54646)
0924 05:02:42.813675 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:42.813700 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:39579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260924 05:02:42.817739 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 17902
2026-09-24T05:02:42Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (860 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225929848825-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:02:42 dist-test-slave-70st krb5kdc[18426](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:02:42 dist-test-slave-70st krb5kdc[18426](info): set up 2 sockets
Sep 24 05:02:42 dist-test-slave-70st krb5kdc[18426](info): commencing operation
krb5kdc: starting...
W20260924 05:02:43.760928 392 mini_kdc.cc:121] Time spent starting KDC: real 0.925s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225929848825-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225929848825-392-0/krb5kdc/test-user.keytab.
Sep 24 05:02:43 dist-test-slave-70st krb5kdc[18426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:02:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:02:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 05:02:43.869684 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225929848825-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225929848825-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225929848825-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39027
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39027
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:02:43.947604 18439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:02:43.947815 18439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:02:43.947837 18439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:02:43.948999 18439 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:02:43.949040 18439 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:02:43.949054 18439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:02:43.949064 18439 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:02:43.949076 18439 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:02:43.950572 18439 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225929848825-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225929848825-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39027
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39027
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18439
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225929848825-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:02:43.950798 18439 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:02:43.951007 18439 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:02:43.955597 18447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:02:43.955621 18445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:02:43.955606 18444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:02:43.955682 18439 server_base.cc:1034] running on GCE node
I20260924 05:02:43.955989 18439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:02:43.956245 18439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:02:43.957401 18439 hybrid_clock.cc:648] HybridClock initialized: now 1790226163957383 us; error 33 us; skew 500 ppm
Sep 24 05:02:43 dist-test-slave-70st krb5kdc[18426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226163, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:02:43.959662 18439 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 05:02:43.960053 18439 webserver.cc:460] Webserver started at http://127.0.98.62:42621/ using document root <none> and password file <none>
I20260924 05:02:43.960242 18439 fs_manager.cc:359] Metadata directory not provided
I20260924 05:02:43.960283 18439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:02:43.960398 18439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:02:43.961403 18439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225929848825-392-0/minicluster-data/master-0/data/instance:
uuid: "7e6e7be3cb9840859de446c5008f6de5"
format_stamp: "Formatted at 2026-09-24 05:02:43 on dist-test-slave-70st"
server_key: "a7592c394ebf95143bfb18f2efa7cbd2"
server_key_iv: "26c6dd5087d90ce344445711213bba24"
server_key_version: "encryptionkey@0"
I20260924 05:02:43.961773 18439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225929848825-392-0/minicluster-data/master-0/wal/instance:
uuid: "7e6e7be3cb9840859de446c5008f6de5"
format_stamp: "Formatted at 2026-09-24 05:02:43 on dist-test-slave-70st"
server_key: "a7592c394ebf95143bfb18f2efa7cbd2"
server_key_iv: "26c6dd5087d90ce344445711213bba24"
server_key_version: "encryptionkey@0"
I20260924 05:02:43.962944 18439 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 05:02:43.963616 18454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:02:43.963763 18439 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 05:02:43.963847 18439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225929848825-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225929848825-392-0/minicluster-data/master-0/wal
uuid: "7e6e7be3cb9840859de446c5008f6de5"
format_stamp: "Formatted at 2026-09-24 05:02:43 on dist-test-slave-70st"
server_key: "a7592c394ebf95143bfb18f2efa7cbd2"
server_key_iv: "26c6dd5087d90ce344445711213bba24"
server_key_version: "encryptionkey@0"
I20260924 05:02:43.963901 18439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225929848825-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225929848825-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225929848825-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:02:43.972169 18439 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:02:43.974159 18439 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:02:43.974263 18439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:02:43.977797 18439 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39027
I20260924 05:02:43.977825 18506 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39027 every 8 connection(s)
I20260924 05:02:43.978138 18439 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:02:43.978900 18507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:02:43.981292 18507 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e6e7be3cb9840859de446c5008f6de5: Bootstrap starting.
I20260924 05:02:43.981905 18507 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e6e7be3cb9840859de446c5008f6de5: Neither blocks nor log segments found. Creating new log.
I20260924 05:02:43.982142 18507 log.cc:824] T 00000000000000000000000000000000 P 7e6e7be3cb9840859de446c5008f6de5: Log is configured to *not* fsync() on all Append() calls
I20260924 05:02:43.982859 18507 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e6e7be3cb9840859de446c5008f6de5: No bootstrap required, opened a new log
I20260924 05:02:43.983722 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 18439
I20260924 05:02:43.983834 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225929848825-392-0/minicluster-data/master-0/wal/instance
I20260924 05:02:43.984015 392 external_mini_cluster.cc:1428] Setting key 8d7306136495bf3e11d132d8c58de1f8
I20260924 05:02:43.984642 18507 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e6e7be3cb9840859de446c5008f6de5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e6e7be3cb9840859de446c5008f6de5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39027 } }
I20260924 05:02:43.984807 18507 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7e6e7be3cb9840859de446c5008f6de5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:02:43.984843 18507 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e6e7be3cb9840859de446c5008f6de5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e6e7be3cb9840859de446c5008f6de5, State: Initialized, Role: FOLLOWER
I20260924 05:02:43.984980 18507 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e6e7be3cb9840859de446c5008f6de5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e6e7be3cb9840859de446c5008f6de5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39027 } }
I20260924 05:02:43.985059 18507 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7e6e7be3cb9840859de446c5008f6de5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:02:43.985087 18507 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e6e7be3cb9840859de446c5008f6de5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:02:43.985154 18507 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e6e7be3cb9840859de446c5008f6de5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:02:43.985888 18507 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e6e7be3cb9840859de446c5008f6de5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e6e7be3cb9840859de446c5008f6de5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39027 } }
I20260924 05:02:43.985975 18507 leader_election.cc:302] T 00000000000000000000000000000000 P 7e6e7be3cb9840859de446c5008f6de5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7e6e7be3cb9840859de446c5008f6de5; no voters:
I20260924 05:02:43.986125 18507 leader_election.cc:288] T 00000000000000000000000000000000 P 7e6e7be3cb9840859de446c5008f6de5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:02:43.986153 18512 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e6e7be3cb9840859de446c5008f6de5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:02:43.986258 18512 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7e6e7be3cb9840859de446c5008f6de5 [term 1 LEADER]: Becoming Leader. State: Replica: 7e6e7be3cb9840859de446c5008f6de5, State: Running, Role: LEADER
I20260924 05:02:43.986382 18507 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e6e7be3cb9840859de446c5008f6de5 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:02:43.986387 18512 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e6e7be3cb9840859de446c5008f6de5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e6e7be3cb9840859de446c5008f6de5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39027 } }
I20260924 05:02:43.986721 18512 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e6e7be3cb9840859de446c5008f6de5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e6e7be3cb9840859de446c5008f6de5. Latest consensus state: current_term: 1 leader_uuid: "7e6e7be3cb9840859de446c5008f6de5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e6e7be3cb9840859de446c5008f6de5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39027 } } }
I20260924 05:02:43.986799 18512 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e6e7be3cb9840859de446c5008f6de5 [sys.catalog]: This master's current role is: LEADER
I20260924 05:02:43.986752 18514 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e6e7be3cb9840859de446c5008f6de5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7e6e7be3cb9840859de446c5008f6de5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e6e7be3cb9840859de446c5008f6de5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39027 } } }
I20260924 05:02:43.986984 18517 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:02:43.986982 18514 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e6e7be3cb9840859de446c5008f6de5 [sys.catalog]: This master's current role is: LEADER
I20260924 05:02:43.987496 18517 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:02:43.989056 18517 catalog_manager.cc:1347] Generated new cluster ID: 72e4a549fb30476bb5e07509225e53e2
I20260924 05:02:43.989115 18517 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 05:02:43 dist-test-slave-70st krb5kdc[18426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:02:43.993909 18511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:43.984561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58054 (local address 127.0.98.62:39027)
0924 05:02:43.984736 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 05:02:43.984745 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 05:02:43.984771 (+ 26us) server_negotiation.cc:407] Connection header received
0924 05:02:43.984876 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:02:43.984883 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:02:43.984943 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:02:43.985072 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:02:43.985823 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:43.987019 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:43.988183 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:43.988767 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:43.988878 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:43.991054 (+ 2176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:02:43.991058 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:02:43.991058 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:02:43.991070 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:02:43.992519 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:02:43.992825 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:02:43.992828 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:02:43.992829 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:02:43.992925 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:02:43.993091 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:02:43.993092 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:02:43.993092 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:02:43.993318 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:02:43.993483 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0924 05:02:43.993583 (+ 100us) server_negotiation.cc:299] Negotiation successful
0924 05:02:43.993710 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":78,"threads_started":1}
I20260924 05:02:43.999291 18517 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:02:43.999981 18517 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:02:44.012148 18517 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7e6e7be3cb9840859de446c5008f6de5: Generated new TSK 0
I20260924 05:02:44.012373 18517 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225929848825-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225929848825-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 05:02:44.036106 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225929848825-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225929848825-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225929848825-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225929848825-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39027
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225929848825-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:02:44.118690 18533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:02:44.118901 18533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:02:44.118923 18533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:02:44.120059 18533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:02:44.120100 18533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:02:44.120126 18533 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 05:02:44.121627 18533 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225929848825-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225929848825-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225929848825-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225929848825-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39027
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18533
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225929848825-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:02:44.121855 18533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:02:44.122061 18533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:02:44.126696 18539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:02:44.126703 18538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:02:44.126710 18541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:02:44.126823 18533 server_base.cc:1034] running on GCE node
I20260924 05:02:44.127022 18533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:02:44.127235 18533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:02:44.128436 18533 hybrid_clock.cc:648] HybridClock initialized: now 1790226164128417 us; error 35 us; skew 500 ppm
Sep 24 05:02:44 dist-test-slave-70st krb5kdc[18426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226164, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:02:44.130888 18533 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 05:02:44.131306 18533 webserver.cc:460] Webserver started at http://127.0.98.1:36395/ using document root <none> and password file <none>
I20260924 05:02:44.131501 18533 fs_manager.cc:359] Metadata directory not provided
I20260924 05:02:44.131541 18533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:02:44.131629 18533 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:02:44.132596 18533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225929848825-392-0/minicluster-data/ts-0/data/instance:
uuid: "fa0cc4f0f5064330b92de104a5a527af"
format_stamp: "Formatted at 2026-09-24 05:02:44 on dist-test-slave-70st"
server_key: "266f7707bba205ed348036adb9dcc881"
server_key_iv: "929ae087b7a197945008713576d82e81"
server_key_version: "encryptionkey@0"
I20260924 05:02:44.132874 18533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225929848825-392-0/minicluster-data/ts-0/wal/instance:
uuid: "fa0cc4f0f5064330b92de104a5a527af"
format_stamp: "Formatted at 2026-09-24 05:02:44 on dist-test-slave-70st"
server_key: "266f7707bba205ed348036adb9dcc881"
server_key_iv: "929ae087b7a197945008713576d82e81"
server_key_version: "encryptionkey@0"
I20260924 05:02:44.134068 18533 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 05:02:44.134713 18548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:02:44.134869 18533 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 05:02:44.134948 18533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225929848825-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225929848825-392-0/minicluster-data/ts-0/wal
uuid: "fa0cc4f0f5064330b92de104a5a527af"
format_stamp: "Formatted at 2026-09-24 05:02:44 on dist-test-slave-70st"
server_key: "266f7707bba205ed348036adb9dcc881"
server_key_iv: "929ae087b7a197945008713576d82e81"
server_key_version: "encryptionkey@0"
I20260924 05:02:44.135016 18533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225929848825-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225929848825-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225929848825-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:02:44.150840 18533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:02:44.152701 18533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:02:44.152843 18533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:02:44.153046 18533 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:02:44.153333 18533 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:02:44.153366 18533 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:02:44.153394 18533 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:02:44.153441 18533 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:02:44.158834 18533 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38253
I20260924 05:02:44.158903 18661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38253 every 8 connection(s)
I20260924 05:02:44.159171 18533 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225929848825-392-0/minicluster-data/ts-0/data/info.pb
I20260924 05:02:44.164970 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 18533
I20260924 05:02:44.165092 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225929848825-392-0/minicluster-data/ts-0/wal/instance
Sep 24 05:02:44 dist-test-slave-70st krb5kdc[18426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226164, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:02:44.165294 392 external_mini_cluster.cc:1428] Setting key 0c455d2d91882fc71eaa1c8793f6e2ab
I20260924 05:02:44.168525 18511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:44.159848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50505 (local address 127.0.98.62:39027)
0924 05:02:44.159888 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 05:02:44.159890 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:02:44.160089 (+ 199us) server_negotiation.cc:407] Connection header received
0924 05:02:44.161034 (+ 945us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:02:44.161036 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:02:44.161051 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:02:44.161112 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:02:44.161790 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:44.162288 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:44.163271 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:44.163475 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:44.163530 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:44.165787 (+ 2257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:02:44.165789 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:02:44.165789 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:02:44.165797 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:02:44.167117 (+ 1320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:02:44.167579 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:02:44.167580 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:02:44.167580 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:02:44.167645 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:02:44.167849 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:02:44.167850 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:02:44.167850 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:02:44.167977 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:02:44.168041 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 05:02:44.168324 (+ 283us) server_negotiation.cc:299] Negotiation successful
0924 05:02:44.168428 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 05:02:44.168606 18664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:44.159837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39027 (local address 127.0.98.1:50505)
0924 05:02:44.159974 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:44.160009 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 05:02:44.160928 (+ 919us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:02:44.161192 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:02:44.161196 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:02:44.161500 (+ 304us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:02:44.161701 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:44.161706 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:44.162370 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:44.162371 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:44.163191 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:44.163193 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:44.163545 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:44.163546 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:44.163664 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:44.164126 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:02:44.164133 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:02:44.165704 (+ 1571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:02:44.167216 (+ 1512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:02:44.167225 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:02:44.167228 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:02:44.167516 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:02:44.167705 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:02:44.167705 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:02:44.167706 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:02:44.167797 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:02:44.168044 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:02:44.168049 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:02:44.168173 (+ 124us) client_negotiation.cc:769] Sending connection context
0924 05:02:44.168292 (+ 119us) client_negotiation.cc:240] Negotiation successful
0924 05:02:44.168428 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":61,"threads_started":1}
I20260924 05:02:44.169077 18662 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39027
I20260924 05:02:44.169169 18662 heartbeater.cc:464] Registering TS with master...
I20260924 05:02:44.169340 18662 heartbeater.cc:511] Master 127.0.98.62:39027 requested a full tablet report, sending...
I20260924 05:02:44.169970 18471 ts_manager.cc:194] Registered new tserver with Master: fa0cc4f0f5064330b92de104a5a527af (127.0.98.1:38253)
I20260924 05:02:44.170882 18471 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:50505
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225929848825-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225929848825-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 05:02:44.189771 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225929848825-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225929848825-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225929848825-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225929848825-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39027
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225929848825-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:02:44.268347 18667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:02:44.268554 18667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:02:44.268577 18667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:02:44.269733 18667 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:02:44.269774 18667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:02:44.269800 18667 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 05:02:44.271265 18667 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225929848825-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225929848825-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225929848825-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225929848825-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39027
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18667
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225929848825-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:02:44.271486 18667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:02:44.271692 18667 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:02:44.276399 18672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:02:44.276417 18675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:02:44.276594 18667 server_base.cc:1034] running on GCE node
W20260924 05:02:44.276674 18673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:02:44.276883 18667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:02:44.277127 18667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:02:44.278259 18667 hybrid_clock.cc:648] HybridClock initialized: now 1790226164278248 us; error 34 us; skew 500 ppm
Sep 24 05:02:44 dist-test-slave-70st krb5kdc[18426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226164, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:02:44.280439 18667 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 05:02:44.280857 18667 webserver.cc:460] Webserver started at http://127.0.98.2:46851/ using document root <none> and password file <none>
I20260924 05:02:44.281055 18667 fs_manager.cc:359] Metadata directory not provided
I20260924 05:02:44.281102 18667 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:02:44.281222 18667 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:02:44.282240 18667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225929848825-392-0/minicluster-data/ts-1/data/instance:
uuid: "1620d90c9db4480f8d9c00614916b5ae"
format_stamp: "Formatted at 2026-09-24 05:02:44 on dist-test-slave-70st"
server_key: "7daeb0c7d2d8ea96f4da47a859bf002c"
server_key_iv: "4641c2a1ef6d1afdabea25dae0179ef9"
server_key_version: "encryptionkey@0"
I20260924 05:02:44.282559 18667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225929848825-392-0/minicluster-data/ts-1/wal/instance:
uuid: "1620d90c9db4480f8d9c00614916b5ae"
format_stamp: "Formatted at 2026-09-24 05:02:44 on dist-test-slave-70st"
server_key: "7daeb0c7d2d8ea96f4da47a859bf002c"
server_key_iv: "4641c2a1ef6d1afdabea25dae0179ef9"
server_key_version: "encryptionkey@0"
I20260924 05:02:44.283806 18667 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 05:02:44.284482 18682 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:02:44.284624 18667 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 05:02:44.284699 18667 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225929848825-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225929848825-392-0/minicluster-data/ts-1/wal
uuid: "1620d90c9db4480f8d9c00614916b5ae"
format_stamp: "Formatted at 2026-09-24 05:02:44 on dist-test-slave-70st"
server_key: "7daeb0c7d2d8ea96f4da47a859bf002c"
server_key_iv: "4641c2a1ef6d1afdabea25dae0179ef9"
server_key_version: "encryptionkey@0"
I20260924 05:02:44.284765 18667 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225929848825-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225929848825-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225929848825-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:02:44.303252 18667 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:02:44.305006 18667 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:02:44.305122 18667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:02:44.305334 18667 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:02:44.305677 18667 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:02:44.305711 18667 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:02:44.305733 18667 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:02:44.305747 18667 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:02:44.311081 18667 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36139
I20260924 05:02:44.311143 18795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36139 every 8 connection(s)
I20260924 05:02:44.311416 18667 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225929848825-392-0/minicluster-data/ts-1/data/info.pb
I20260924 05:02:44.314152 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 18667
I20260924 05:02:44.314260 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225929848825-392-0/minicluster-data/ts-1/wal/instance
I20260924 05:02:44.314426 392 external_mini_cluster.cc:1428] Setting key 57849aedf8f2c0bcdef06d8273952a06
Sep 24 05:02:44 dist-test-slave-70st krb5kdc[18426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226164, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:02:44.320991 18511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:44.312137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54427 (local address 127.0.98.62:39027)
0924 05:02:44.312202 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 05:02:44.312205 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:02:44.312357 (+ 152us) server_negotiation.cc:407] Connection header received
0924 05:02:44.313274 (+ 917us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:02:44.313275 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:02:44.313288 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:02:44.313358 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:02:44.313942 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:44.314539 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:44.315834 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:44.316028 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:44.316081 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:44.318232 (+ 2151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:02:44.318233 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:02:44.318233 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:02:44.318240 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:02:44.319569 (+ 1329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:02:44.320021 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:02:44.320022 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:02:44.320022 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:02:44.320071 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:02:44.320328 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:02:44.320328 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:02:44.320328 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:02:44.320445 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:02:44.320491 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 05:02:44.320836 (+ 345us) server_negotiation.cc:299] Negotiation successful
0924 05:02:44.320921 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 05:02:44.321183 18798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:44.312091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39027 (local address 127.0.98.2:54427)
0924 05:02:44.312239 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:44.312278 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 05:02:44.313158 (+ 880us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:02:44.313438 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:02:44.313441 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:02:44.313680 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:02:44.313874 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:44.313878 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:44.314623 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:44.314624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:44.315753 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:44.315756 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:44.316097 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:44.316098 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:44.316232 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:44.316753 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:02:44.316761 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:02:44.318151 (+ 1390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:02:44.319647 (+ 1496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:02:44.319654 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:02:44.319657 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:02:44.319953 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:02:44.320126 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:02:44.320127 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:02:44.320127 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:02:44.320271 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:02:44.320499 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:02:44.320505 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:02:44.320669 (+ 164us) client_negotiation.cc:769] Sending connection context
0924 05:02:44.320794 (+ 125us) client_negotiation.cc:240] Negotiation successful
0924 05:02:44.320959 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":76,"threads_started":1}
I20260924 05:02:44.321698 18796 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39027
I20260924 05:02:44.321790 18796 heartbeater.cc:464] Registering TS with master...
I20260924 05:02:44.321955 18796 heartbeater.cc:511] Master 127.0.98.62:39027 requested a full tablet report, sending...
I20260924 05:02:44.322352 18471 ts_manager.cc:194] Registered new tserver with Master: 1620d90c9db4480f8d9c00614916b5ae (127.0.98.2:36139)
I20260924 05:02:44.322700 18471 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:54427
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225929848825-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225929848825-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 05:02:44.339708 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225929848825-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225929848825-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225929848825-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225929848825-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39027
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225929848825-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:02:44.417896 18801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:02:44.418107 18801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:02:44.418138 18801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:02:44.419301 18801 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:02:44.419341 18801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:02:44.419375 18801 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 05:02:44.420806 18801 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225929848825-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225929848825-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225929848825-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225929848825-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39027
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18801
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225929848825-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:02:44.421027 18801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:02:44.421224 18801 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:02:44.425925 18807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:02:44.425922 18809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:02:44.425971 18806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:02:44.425961 18801 server_base.cc:1034] running on GCE node
I20260924 05:02:44.426298 18801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:02:44.426517 18801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:02:44.427665 18801 hybrid_clock.cc:648] HybridClock initialized: now 1790226164427648 us; error 33 us; skew 500 ppm
Sep 24 05:02:44 dist-test-slave-70st krb5kdc[18426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226164, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:02:44.430099 18801 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 05:02:44.430538 18801 webserver.cc:460] Webserver started at http://127.0.98.3:36345/ using document root <none> and password file <none>
I20260924 05:02:44.430722 18801 fs_manager.cc:359] Metadata directory not provided
I20260924 05:02:44.430770 18801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:02:44.430888 18801 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:02:44.431874 18801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225929848825-392-0/minicluster-data/ts-2/data/instance:
uuid: "7c775be5d37b4e1bb83ee90efa43d752"
format_stamp: "Formatted at 2026-09-24 05:02:44 on dist-test-slave-70st"
server_key: "de464eb6bb47c63875ed837e0794a408"
server_key_iv: "380ead2d316b78a5ba247feace8587ab"
server_key_version: "encryptionkey@0"
I20260924 05:02:44.432183 18801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225929848825-392-0/minicluster-data/ts-2/wal/instance:
uuid: "7c775be5d37b4e1bb83ee90efa43d752"
format_stamp: "Formatted at 2026-09-24 05:02:44 on dist-test-slave-70st"
server_key: "de464eb6bb47c63875ed837e0794a408"
server_key_iv: "380ead2d316b78a5ba247feace8587ab"
server_key_version: "encryptionkey@0"
I20260924 05:02:44.433386 18801 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 05:02:44.434154 18816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:02:44.434332 18801 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 05:02:44.434415 18801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225929848825-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225929848825-392-0/minicluster-data/ts-2/wal
uuid: "7c775be5d37b4e1bb83ee90efa43d752"
format_stamp: "Formatted at 2026-09-24 05:02:44 on dist-test-slave-70st"
server_key: "de464eb6bb47c63875ed837e0794a408"
server_key_iv: "380ead2d316b78a5ba247feace8587ab"
server_key_version: "encryptionkey@0"
I20260924 05:02:44.434474 18801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225929848825-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225929848825-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225929848825-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:02:44.445322 18801 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:02:44.447433 18801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:02:44.447575 18801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:02:44.447786 18801 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:02:44.448068 18801 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:02:44.448100 18801 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:02:44.448128 18801 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:02:44.448150 18801 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:02:44.453611 18801 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39699
I20260924 05:02:44.453670 18929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39699 every 8 connection(s)
I20260924 05:02:44.453949 18801 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225929848825-392-0/minicluster-data/ts-2/data/info.pb
I20260924 05:02:44.454140 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 18801
I20260924 05:02:44.454247 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225929848825-392-0/minicluster-data/ts-2/wal/instance
I20260924 05:02:44.454430 392 external_mini_cluster.cc:1428] Setting key f46c649c916dec125fc7a9542dbe8e22
Sep 24 05:02:44 dist-test-slave-70st krb5kdc[18426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226164, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:02:44.462929 18511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:44.454797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35589 (local address 127.0.98.62:39027)
0924 05:02:44.454837 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 05:02:44.454839 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:02:44.455106 (+ 267us) server_negotiation.cc:407] Connection header received
0924 05:02:44.456137 (+ 1031us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:02:44.456138 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:02:44.456148 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:02:44.456197 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:02:44.456791 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:44.457309 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:44.458437 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:44.458626 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:44.458687 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:44.460636 (+ 1949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:02:44.460638 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:02:44.460638 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:02:44.460645 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:02:44.461913 (+ 1268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:02:44.462257 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:02:44.462258 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:02:44.462259 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:02:44.462304 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:02:44.462484 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:02:44.462485 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:02:44.462485 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:02:44.462574 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:02:44.462617 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 05:02:44.462814 (+ 197us) server_negotiation.cc:299] Negotiation successful
0924 05:02:44.462866 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 05:02:44.463114 18932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:44.454792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39027 (local address 127.0.98.3:35589)
0924 05:02:44.454935 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:44.454991 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 05:02:44.456035 (+ 1044us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:02:44.456258 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:02:44.456261 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:02:44.456507 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:02:44.456700 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:44.456704 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:44.457405 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:44.457406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:44.458358 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:44.458360 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:44.458691 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:44.458692 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:44.458790 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:44.459226 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:02:44.459235 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:02:44.460546 (+ 1311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:02:44.461976 (+ 1430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:02:44.461982 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:02:44.461983 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:02:44.462181 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:02:44.462350 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:02:44.462350 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:02:44.462350 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:02:44.462441 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:02:44.462614 (+ 173us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:02:44.462616 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:02:44.462721 (+ 105us) client_negotiation.cc:769] Sending connection context
0924 05:02:44.462796 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 05:02:44.462906 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":65,"threads_started":1}
I20260924 05:02:44.463615 18930 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39027
I20260924 05:02:44.463709 18930 heartbeater.cc:464] Registering TS with master...
I20260924 05:02:44.463876 18930 heartbeater.cc:511] Master 127.0.98.62:39027 requested a full tablet report, sending...
I20260924 05:02:44.464228 18471 ts_manager.cc:194] Registered new tserver with Master: 7c775be5d37b4e1bb83ee90efa43d752 (127.0.98.3:39699)
I20260924 05:02:44.464607 18471 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35589
I20260924 05:02:44.466364 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 05:02:44.472862 18511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:44.467663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58058 (local address 127.0.98.62:39027)
0924 05:02:44.467723 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 05:02:44.467725 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:02:44.467822 (+ 97us) server_negotiation.cc:407] Connection header received
0924 05:02:44.467858 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:02:44.467859 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:02:44.467867 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:02:44.467908 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:02:44.468373 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:44.468826 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:44.469512 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:44.469684 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:44.469728 (+ 44us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:44.470776 (+ 1048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:02:44.470778 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:02:44.470778 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:02:44.470783 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:02:44.471942 (+ 1159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:02:44.472290 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:02:44.472291 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:02:44.472291 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:02:44.472337 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:02:44.472488 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:02:44.472488 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:02:44.472488 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:02:44.472597 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:02:44.472669 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 05:02:44.472733 (+ 64us) server_negotiation.cc:299] Negotiation successful
0924 05:02:44.472779 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 05:02:44.474298 18471 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58058:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 05:02:44.475750 18471 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 05:02:44.481562 18938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:44.477616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36139 (local address 127.0.0.1:42158)
0924 05:02:44.477807 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:44.477847 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 05:02:44.477908 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:02:44.478244 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:02:44.478247 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:02:44.478249 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:02:44.478357 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:44.478360 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:44.479670 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:44.479671 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:44.480408 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:44.480411 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:44.481244 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:44.481245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:44.481323 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:44.481337 (+ 14us) client_negotiation.cc:769] Sending connection context
0924 05:02:44.481408 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 05:02:44.481488 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":79,"threads_started":1}
I20260924 05:02:44.481565 18943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:44.477914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42158 (local address 127.0.98.2:36139)
0924 05:02:44.478068 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 05:02:44.478071 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:02:44.478088 (+ 17us) server_negotiation.cc:407] Connection header received
0924 05:02:44.478127 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:02:44.478129 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:02:44.478145 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:02:44.478224 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:02:44.478442 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:44.479561 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:44.480498 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:44.481174 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:44.481243 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:44.481299 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 05:02:44.481446 (+ 147us) server_negotiation.cc:299] Negotiation successful
0924 05:02:44.481488 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":84,"threads_started":1}
I20260924 05:02:44.481860 18941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:44.477758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36022 (local address 127.0.98.3:39699)
0924 05:02:44.477930 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 05:02:44.477935 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 05:02:44.478176 (+ 241us) server_negotiation.cc:407] Connection header received
0924 05:02:44.478214 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:02:44.478216 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:02:44.478229 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:02:44.478315 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:02:44.478679 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:44.479351 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:44.480720 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:44.481409 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:44.481502 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:44.481558 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 05:02:44.481743 (+ 185us) server_negotiation.cc:299] Negotiation successful
0924 05:02:44.481788 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":74,"threads_started":1}
I20260924 05:02:44.482074 18940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:44.477625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39699 (local address 127.0.0.1:36022)
0924 05:02:44.478079 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:44.478087 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 05:02:44.478136 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:02:44.478562 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:02:44.478563 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:02:44.478565 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:02:44.478594 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:44.478595 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:44.479438 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:44.479439 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:44.480649 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:44.480651 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:44.481591 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:44.481593 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:44.481652 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:44.481655 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:02:44.481961 (+ 306us) client_negotiation.cc:240] Negotiation successful
0924 05:02:44.482004 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"mutex_wait_us":4,"thread_start_us":72,"threads_started":1}
I20260924 05:02:44.482264 18730 tablet_service.cc:1467] Processing CreateTablet for tablet f20fe4258e62426c8d948cbd456f4542 (DEFAULT_TABLE table=test-table [id=4adceb1421de4654bc40aec054b53f67]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:02:44.482530 18730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f20fe4258e62426c8d948cbd456f4542. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:02:44.482561 18942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:44.477893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49542 (local address 127.0.98.1:38253)
0924 05:02:44.478045 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 05:02:44.478050 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 05:02:44.478270 (+ 220us) server_negotiation.cc:407] Connection header received
0924 05:02:44.478312 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:02:44.478314 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:02:44.478330 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:02:44.478408 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:02:44.478518 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:44.479535 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:44.480908 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:44.481585 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:44.482352 (+ 767us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:44.482423 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 05:02:44.482448 (+ 25us) server_negotiation.cc:299] Negotiation successful
0924 05:02:44.482490 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":75,"threads_started":1}
I20260924 05:02:44.482658 18939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:44.477625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38253 (local address 127.0.0.1:49542)
0924 05:02:44.478182 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:44.478188 (+ 6us) client_negotiation.cc:174] Beginning negotiation
0924 05:02:44.478222 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:02:44.478417 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:02:44.478419 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:02:44.478421 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:02:44.478462 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:44.478464 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:44.479630 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:44.479631 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:44.480843 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:44.480845 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:44.481981 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:44.481982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:44.482043 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:44.482047 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 05:02:44.482084 (+ 37us) client_negotiation.cc:240] Negotiation successful
0924 05:02:44.482116 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"mutex_wait_us":27,"thread_start_us":71,"threads_started":1}
I20260924 05:02:44.482721 18864 tablet_service.cc:1467] Processing CreateTablet for tablet f20fe4258e62426c8d948cbd456f4542 (DEFAULT_TABLE table=test-table [id=4adceb1421de4654bc40aec054b53f67]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:02:44.483040 18864 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f20fe4258e62426c8d948cbd456f4542. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:02:44.483276 18596 tablet_service.cc:1467] Processing CreateTablet for tablet f20fe4258e62426c8d948cbd456f4542 (DEFAULT_TABLE table=test-table [id=4adceb1421de4654bc40aec054b53f67]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:02:44.483542 18596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f20fe4258e62426c8d948cbd456f4542. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:02:44.484984 18944 tablet_bootstrap.cc:492] T f20fe4258e62426c8d948cbd456f4542 P 1620d90c9db4480f8d9c00614916b5ae: Bootstrap starting.
I20260924 05:02:44.485582 18944 tablet_bootstrap.cc:654] T f20fe4258e62426c8d948cbd456f4542 P 1620d90c9db4480f8d9c00614916b5ae: Neither blocks nor log segments found. Creating new log.
I20260924 05:02:44.486069 18945 tablet_bootstrap.cc:492] T f20fe4258e62426c8d948cbd456f4542 P 7c775be5d37b4e1bb83ee90efa43d752: Bootstrap starting.
I20260924 05:02:44.486107 18944 log.cc:824] T f20fe4258e62426c8d948cbd456f4542 P 1620d90c9db4480f8d9c00614916b5ae: Log is configured to *not* fsync() on all Append() calls
I20260924 05:02:44.486501 18947 tablet_bootstrap.cc:492] T f20fe4258e62426c8d948cbd456f4542 P fa0cc4f0f5064330b92de104a5a527af: Bootstrap starting.
I20260924 05:02:44.486785 18945 tablet_bootstrap.cc:654] T f20fe4258e62426c8d948cbd456f4542 P 7c775be5d37b4e1bb83ee90efa43d752: Neither blocks nor log segments found. Creating new log.
I20260924 05:02:44.487190 18944 tablet_bootstrap.cc:492] T f20fe4258e62426c8d948cbd456f4542 P 1620d90c9db4480f8d9c00614916b5ae: No bootstrap required, opened a new log
I20260924 05:02:44.487267 18944 ts_tablet_manager.cc:1397] T f20fe4258e62426c8d948cbd456f4542 P 1620d90c9db4480f8d9c00614916b5ae: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 05:02:44.487377 18945 log.cc:824] T f20fe4258e62426c8d948cbd456f4542 P 7c775be5d37b4e1bb83ee90efa43d752: Log is configured to *not* fsync() on all Append() calls
I20260924 05:02:44.488214 18945 tablet_bootstrap.cc:492] T f20fe4258e62426c8d948cbd456f4542 P 7c775be5d37b4e1bb83ee90efa43d752: No bootstrap required, opened a new log
I20260924 05:02:44.488277 18945 ts_tablet_manager.cc:1397] T f20fe4258e62426c8d948cbd456f4542 P 7c775be5d37b4e1bb83ee90efa43d752: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 05:02:44.489100 18944 raft_consensus.cc:348] T f20fe4258e62426c8d948cbd456f4542 P 1620d90c9db4480f8d9c00614916b5ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1620d90c9db4480f8d9c00614916b5ae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36139 } } peers { permanent_uuid: "7c775be5d37b4e1bb83ee90efa43d752" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39699 } } peers { permanent_uuid: "fa0cc4f0f5064330b92de104a5a527af" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38253 } }
I20260924 05:02:44.489288 18944 raft_consensus.cc:374] T f20fe4258e62426c8d948cbd456f4542 P 1620d90c9db4480f8d9c00614916b5ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:02:44.489331 18944 raft_consensus.cc:729] T f20fe4258e62426c8d948cbd456f4542 P 1620d90c9db4480f8d9c00614916b5ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1620d90c9db4480f8d9c00614916b5ae, State: Initialized, Role: FOLLOWER
I20260924 05:02:44.489468 18944 consensus_queue.cc:260] T f20fe4258e62426c8d948cbd456f4542 P 1620d90c9db4480f8d9c00614916b5ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1620d90c9db4480f8d9c00614916b5ae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36139 } } peers { permanent_uuid: "7c775be5d37b4e1bb83ee90efa43d752" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39699 } } peers { permanent_uuid: "fa0cc4f0f5064330b92de104a5a527af" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38253 } }
I20260924 05:02:44.489660 18947 tablet_bootstrap.cc:654] T f20fe4258e62426c8d948cbd456f4542 P fa0cc4f0f5064330b92de104a5a527af: Neither blocks nor log segments found. Creating new log.
I20260924 05:02:44.489741 18944 ts_tablet_manager.cc:1428] T f20fe4258e62426c8d948cbd456f4542 P 1620d90c9db4480f8d9c00614916b5ae: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260924 05:02:44.489866 18796 heartbeater.cc:503] Master 127.0.98.62:39027 was elected leader, sending a full tablet report...
I20260924 05:02:44.490082 18945 raft_consensus.cc:348] T f20fe4258e62426c8d948cbd456f4542 P 7c775be5d37b4e1bb83ee90efa43d752 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1620d90c9db4480f8d9c00614916b5ae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36139 } } peers { permanent_uuid: "7c775be5d37b4e1bb83ee90efa43d752" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39699 } } peers { permanent_uuid: "fa0cc4f0f5064330b92de104a5a527af" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38253 } }
I20260924 05:02:44.490190 18945 raft_consensus.cc:374] T f20fe4258e62426c8d948cbd456f4542 P 7c775be5d37b4e1bb83ee90efa43d752 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:02:44.490207 18945 raft_consensus.cc:729] T f20fe4258e62426c8d948cbd456f4542 P 7c775be5d37b4e1bb83ee90efa43d752 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c775be5d37b4e1bb83ee90efa43d752, State: Initialized, Role: FOLLOWER
I20260924 05:02:44.490204 18947 log.cc:824] T f20fe4258e62426c8d948cbd456f4542 P fa0cc4f0f5064330b92de104a5a527af: Log is configured to *not* fsync() on all Append() calls
I20260924 05:02:44.490286 18945 consensus_queue.cc:260] T f20fe4258e62426c8d948cbd456f4542 P 7c775be5d37b4e1bb83ee90efa43d752 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1620d90c9db4480f8d9c00614916b5ae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36139 } } peers { permanent_uuid: "7c775be5d37b4e1bb83ee90efa43d752" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39699 } } peers { permanent_uuid: "fa0cc4f0f5064330b92de104a5a527af" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38253 } }
I20260924 05:02:44.490500 18945 ts_tablet_manager.cc:1428] T f20fe4258e62426c8d948cbd456f4542 P 7c775be5d37b4e1bb83ee90efa43d752: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 05:02:44.490604 18930 heartbeater.cc:503] Master 127.0.98.62:39027 was elected leader, sending a full tablet report...
I20260924 05:02:44.490960 18947 tablet_bootstrap.cc:492] T f20fe4258e62426c8d948cbd456f4542 P fa0cc4f0f5064330b92de104a5a527af: No bootstrap required, opened a new log
I20260924 05:02:44.491060 18947 ts_tablet_manager.cc:1397] T f20fe4258e62426c8d948cbd456f4542 P fa0cc4f0f5064330b92de104a5a527af: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260924 05:02:44.491500 18951 raft_consensus.cc:482] T f20fe4258e62426c8d948cbd456f4542 P 7c775be5d37b4e1bb83ee90efa43d752 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 05:02:44.491576 18951 raft_consensus.cc:504] T f20fe4258e62426c8d948cbd456f4542 P 7c775be5d37b4e1bb83ee90efa43d752 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1620d90c9db4480f8d9c00614916b5ae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36139 } } peers { permanent_uuid: "7c775be5d37b4e1bb83ee90efa43d752" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39699 } } peers { permanent_uuid: "fa0cc4f0f5064330b92de104a5a527af" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38253 } }
I20260924 05:02:44.491863 18951 leader_election.cc:288] T f20fe4258e62426c8d948cbd456f4542 P 7c775be5d37b4e1bb83ee90efa43d752 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1620d90c9db4480f8d9c00614916b5ae (127.0.98.2:36139), fa0cc4f0f5064330b92de104a5a527af (127.0.98.1:38253)
I20260924 05:02:44.492872 18947 raft_consensus.cc:348] T f20fe4258e62426c8d948cbd456f4542 P fa0cc4f0f5064330b92de104a5a527af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1620d90c9db4480f8d9c00614916b5ae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36139 } } peers { permanent_uuid: "7c775be5d37b4e1bb83ee90efa43d752" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39699 } } peers { permanent_uuid: "fa0cc4f0f5064330b92de104a5a527af" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38253 } }
I20260924 05:02:44.493038 18947 raft_consensus.cc:374] T f20fe4258e62426c8d948cbd456f4542 P fa0cc4f0f5064330b92de104a5a527af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:02:44.493067 18947 raft_consensus.cc:729] T f20fe4258e62426c8d948cbd456f4542 P fa0cc4f0f5064330b92de104a5a527af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa0cc4f0f5064330b92de104a5a527af, State: Initialized, Role: FOLLOWER
I20260924 05:02:44.493161 18947 consensus_queue.cc:260] T f20fe4258e62426c8d948cbd456f4542 P fa0cc4f0f5064330b92de104a5a527af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1620d90c9db4480f8d9c00614916b5ae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36139 } } peers { permanent_uuid: "7c775be5d37b4e1bb83ee90efa43d752" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39699 } } peers { permanent_uuid: "fa0cc4f0f5064330b92de104a5a527af" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38253 } }
I20260924 05:02:44.493391 18947 ts_tablet_manager.cc:1428] T f20fe4258e62426c8d948cbd456f4542 P fa0cc4f0f5064330b92de104a5a527af: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 05:02:44.493520 18662 heartbeater.cc:503] Master 127.0.98.62:39027 was elected leader, sending a full tablet report...
I20260924 05:02:44.494831 18943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:44.492060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56963 (local address 127.0.98.2:36139)
0924 05:02:44.492148 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0924 05:02:44.492150 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:02:44.492157 (+ 7us) server_negotiation.cc:407] Connection header received
0924 05:02:44.492175 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:02:44.492176 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:02:44.492185 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:02:44.492230 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:02:44.492417 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:44.492908 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:44.494052 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:44.494538 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:44.494595 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:44.494603 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 05:02:44.494735 (+ 132us) server_negotiation.cc:299] Negotiation successful
0924 05:02:44.494766 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 05:02:44.494831 18932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:44.491972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36139 (local address 127.0.98.3:56963)
0924 05:02:44.492017 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:44.492028 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 05:02:44.492062 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:02:44.492244 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:02:44.492245 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:02:44.492246 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:02:44.492348 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:44.492350 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:44.492974 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:44.492975 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:44.493977 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:44.493979 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:44.494602 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:44.494603 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:44.494681 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:44.494683 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 05:02:44.494725 (+ 42us) client_negotiation.cc:240] Negotiation successful
0924 05:02:44.494769 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260924 05:02:44.495142 18750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20fe4258e62426c8d948cbd456f4542" candidate_uuid: "7c775be5d37b4e1bb83ee90efa43d752" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1620d90c9db4480f8d9c00614916b5ae" is_pre_election: true
I20260924 05:02:44.495249 18750 raft_consensus.cc:2454] T f20fe4258e62426c8d948cbd456f4542 P 1620d90c9db4480f8d9c00614916b5ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c775be5d37b4e1bb83ee90efa43d752 in term 0.
I20260924 05:02:44.495277 18942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:44.492102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36905 (local address 127.0.98.1:38253)
0924 05:02:44.492140 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 05:02:44.492142 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:02:44.492353 (+ 211us) server_negotiation.cc:407] Connection header received
0924 05:02:44.492382 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:02:44.492383 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:02:44.492392 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:02:44.492448 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:02:44.492542 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:44.493254 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:44.494539 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:44.494907 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:44.494979 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:44.494988 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 05:02:44.495179 (+ 191us) server_negotiation.cc:299] Negotiation successful
0924 05:02:44.495216 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 05:02:44.495277 18952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:44.491985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38253 (local address 127.0.98.3:36905)
0924 05:02:44.492290 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:44.492299 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:02:44.492346 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:02:44.492448 (+ 102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:02:44.492449 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:02:44.492450 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:02:44.492496 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:44.492497 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:44.493356 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:44.493357 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:44.494474 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:44.494477 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:44.495034 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:44.495035 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:44.495115 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:44.495118 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:02:44.495167 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 05:02:44.495203 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":66,"threads_started":1}
I20260924 05:02:44.495467 18817 leader_election.cc:302] T f20fe4258e62426c8d948cbd456f4542 P 7c775be5d37b4e1bb83ee90efa43d752 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1620d90c9db4480f8d9c00614916b5ae, 7c775be5d37b4e1bb83ee90efa43d752; no voters:
I20260924 05:02:44.495517 18616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20fe4258e62426c8d948cbd456f4542" candidate_uuid: "7c775be5d37b4e1bb83ee90efa43d752" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa0cc4f0f5064330b92de104a5a527af" is_pre_election: true
I20260924 05:02:44.495636 18616 raft_consensus.cc:2454] T f20fe4258e62426c8d948cbd456f4542 P fa0cc4f0f5064330b92de104a5a527af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c775be5d37b4e1bb83ee90efa43d752 in term 0.
I20260924 05:02:44.495736 18951 raft_consensus.cc:2781] T f20fe4258e62426c8d948cbd456f4542 P 7c775be5d37b4e1bb83ee90efa43d752 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 05:02:44.495796 18951 raft_consensus.cc:482] T f20fe4258e62426c8d948cbd456f4542 P 7c775be5d37b4e1bb83ee90efa43d752 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 05:02:44.495820 18951 raft_consensus.cc:3037] T f20fe4258e62426c8d948cbd456f4542 P 7c775be5d37b4e1bb83ee90efa43d752 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:02:44.496407 18951 raft_consensus.cc:504] T f20fe4258e62426c8d948cbd456f4542 P 7c775be5d37b4e1bb83ee90efa43d752 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1620d90c9db4480f8d9c00614916b5ae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36139 } } peers { permanent_uuid: "7c775be5d37b4e1bb83ee90efa43d752" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39699 } } peers { permanent_uuid: "fa0cc4f0f5064330b92de104a5a527af" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38253 } }
I20260924 05:02:44.496511 18951 leader_election.cc:288] T f20fe4258e62426c8d948cbd456f4542 P 7c775be5d37b4e1bb83ee90efa43d752 [CANDIDATE]: Term 1 election: Requested vote from peers 1620d90c9db4480f8d9c00614916b5ae (127.0.98.2:36139), fa0cc4f0f5064330b92de104a5a527af (127.0.98.1:38253)
I20260924 05:02:44.496632 18616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20fe4258e62426c8d948cbd456f4542" candidate_uuid: "7c775be5d37b4e1bb83ee90efa43d752" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa0cc4f0f5064330b92de104a5a527af"
I20260924 05:02:44.496641 18750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20fe4258e62426c8d948cbd456f4542" candidate_uuid: "7c775be5d37b4e1bb83ee90efa43d752" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1620d90c9db4480f8d9c00614916b5ae"
I20260924 05:02:44.496694 18616 raft_consensus.cc:3037] T f20fe4258e62426c8d948cbd456f4542 P fa0cc4f0f5064330b92de104a5a527af [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:02:44.496696 18750 raft_consensus.cc:3037] T f20fe4258e62426c8d948cbd456f4542 P 1620d90c9db4480f8d9c00614916b5ae [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:02:44.497473 18616 raft_consensus.cc:2454] T f20fe4258e62426c8d948cbd456f4542 P fa0cc4f0f5064330b92de104a5a527af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c775be5d37b4e1bb83ee90efa43d752 in term 1.
I20260924 05:02:44.497470 18750 raft_consensus.cc:2454] T f20fe4258e62426c8d948cbd456f4542 P 1620d90c9db4480f8d9c00614916b5ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c775be5d37b4e1bb83ee90efa43d752 in term 1.
I20260924 05:02:44.497610 18817 leader_election.cc:302] T f20fe4258e62426c8d948cbd456f4542 P 7c775be5d37b4e1bb83ee90efa43d752 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1620d90c9db4480f8d9c00614916b5ae, 7c775be5d37b4e1bb83ee90efa43d752; no voters:
I20260924 05:02:44.497681 18951 raft_consensus.cc:2781] T f20fe4258e62426c8d948cbd456f4542 P 7c775be5d37b4e1bb83ee90efa43d752 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:02:44.497794 18951 raft_consensus.cc:686] T f20fe4258e62426c8d948cbd456f4542 P 7c775be5d37b4e1bb83ee90efa43d752 [term 1 LEADER]: Becoming Leader. State: Replica: 7c775be5d37b4e1bb83ee90efa43d752, State: Running, Role: LEADER
I20260924 05:02:44.497865 18951 consensus_queue.cc:237] T f20fe4258e62426c8d948cbd456f4542 P 7c775be5d37b4e1bb83ee90efa43d752 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1620d90c9db4480f8d9c00614916b5ae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36139 } } peers { permanent_uuid: "7c775be5d37b4e1bb83ee90efa43d752" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39699 } } peers { permanent_uuid: "fa0cc4f0f5064330b92de104a5a527af" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38253 } }
I20260924 05:02:44.498540 18471 catalog_manager.cc:5495] T f20fe4258e62426c8d948cbd456f4542 P 7c775be5d37b4e1bb83ee90efa43d752 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7c775be5d37b4e1bb83ee90efa43d752 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "7c775be5d37b4e1bb83ee90efa43d752" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1620d90c9db4480f8d9c00614916b5ae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c775be5d37b4e1bb83ee90efa43d752" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39699 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fa0cc4f0f5064330b92de104a5a527af" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38253 } health_report { overall_health: UNKNOWN } } }
I20260924 05:02:44.502804 18941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:44.500817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36036 (local address 127.0.98.3:39699)
0924 05:02:44.500858 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 05:02:44.500859 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:02:44.500869 (+ 10us) server_negotiation.cc:407] Connection header received
0924 05:02:44.500937 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:02:44.500937 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:02:44.500950 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:02:44.501002 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:02:44.501087 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:44.501595 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:44.502268 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:44.502440 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:44.502479 (+ 39us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:44.502575 (+ 96us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:02:44.502647 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:02:44.502707 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 05:02:44.502725 (+ 18us) server_negotiation.cc:299] Negotiation successful
0924 05:02:44.502731 (+ 6us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 05:02:44.504230 18750 raft_consensus.cc:1261] T f20fe4258e62426c8d948cbd456f4542 P 1620d90c9db4480f8d9c00614916b5ae [term 1 FOLLOWER]: Refusing update from remote peer 7c775be5d37b4e1bb83ee90efa43d752: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:02:44.504247 18616 raft_consensus.cc:1261] T f20fe4258e62426c8d948cbd456f4542 P fa0cc4f0f5064330b92de104a5a527af [term 1 FOLLOWER]: Refusing update from remote peer 7c775be5d37b4e1bb83ee90efa43d752: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:02:44.504537 18951 consensus_queue.cc:1035] T f20fe4258e62426c8d948cbd456f4542 P 7c775be5d37b4e1bb83ee90efa43d752 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1620d90c9db4480f8d9c00614916b5ae" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:02:44.504662 18954 consensus_queue.cc:1035] T f20fe4258e62426c8d948cbd456f4542 P 7c775be5d37b4e1bb83ee90efa43d752 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa0cc4f0f5064330b92de104a5a527af" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:02:44.507182 18958 mvcc.cc:205] Tried to move back new op lower bound from 7332766369807138816 to 7332766369783865344. Current Snapshot: MvccSnapshot[applied={T|T < 7332766369807138816}]
I20260924 05:02:44.507359 18959 mvcc.cc:205] Tried to move back new op lower bound from 7332766369807138816 to 7332766369783865344. Current Snapshot: MvccSnapshot[applied={T|T < 7332766369807138816}]
I20260924 05:02:44.508930 18471 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58058:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:02:44.509030 18471 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58058:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:02:44.509766 18471 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7e6e7be3cb9840859de446c5008f6de5: Sending DeleteTablet for 3 replicas of tablet f20fe4258e62426c8d948cbd456f4542
I20260924 05:02:44.510051 18730 tablet_service.cc:1514] Processing DeleteTablet for tablet f20fe4258e62426c8d948cbd456f4542 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:02:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42158
I20260924 05:02:44.510075 18596 tablet_service.cc:1514] Processing DeleteTablet for tablet f20fe4258e62426c8d948cbd456f4542 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:02:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49542
I20260924 05:02:44.510063 18864 tablet_service.cc:1514] Processing DeleteTablet for tablet f20fe4258e62426c8d948cbd456f4542 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:02:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36022
I20260924 05:02:44.510284 18967 tablet_replica.cc:330] T f20fe4258e62426c8d948cbd456f4542 P 1620d90c9db4480f8d9c00614916b5ae: stopping tablet replica
I20260924 05:02:44.510316 18968 tablet_replica.cc:330] T f20fe4258e62426c8d948cbd456f4542 P fa0cc4f0f5064330b92de104a5a527af: stopping tablet replica
I20260924 05:02:44.510402 18968 raft_consensus.cc:2229] T f20fe4258e62426c8d948cbd456f4542 P fa0cc4f0f5064330b92de104a5a527af [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:02:44.510402 18967 raft_consensus.cc:2229] T f20fe4258e62426c8d948cbd456f4542 P 1620d90c9db4480f8d9c00614916b5ae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:02:44.510499 18968 raft_consensus.cc:2258] T f20fe4258e62426c8d948cbd456f4542 P fa0cc4f0f5064330b92de104a5a527af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:02:44.510509 18967 raft_consensus.cc:2258] T f20fe4258e62426c8d948cbd456f4542 P 1620d90c9db4480f8d9c00614916b5ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:02:44.510640 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 18533
I20260924 05:02:44.511690 18967 ts_tablet_manager.cc:1905] T f20fe4258e62426c8d948cbd456f4542 P 1620d90c9db4480f8d9c00614916b5ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:02:44.513221 18967 ts_tablet_manager.cc:1918] T f20fe4258e62426c8d948cbd456f4542 P 1620d90c9db4480f8d9c00614916b5ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 05:02:44.513279 18967 log.cc:1196] T f20fe4258e62426c8d948cbd456f4542 P 1620d90c9db4480f8d9c00614916b5ae: Deleting WAL directory at /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225929848825-392-0/minicluster-data/ts-1/wal/wals/f20fe4258e62426c8d948cbd456f4542
I20260924 05:02:44.513535 18967 ts_tablet_manager.cc:1939] T f20fe4258e62426c8d948cbd456f4542 P 1620d90c9db4480f8d9c00614916b5ae: Deleting consensus metadata
I20260924 05:02:44.513898 18455 catalog_manager.cc:4841] TS 1620d90c9db4480f8d9c00614916b5ae (127.0.98.2:36139): tablet f20fe4258e62426c8d948cbd456f4542 (table test-table [id=4adceb1421de4654bc40aec054b53f67]) successfully deleted
I20260924 05:02:44.514043 18969 tablet_replica.cc:330] T f20fe4258e62426c8d948cbd456f4542 P 7c775be5d37b4e1bb83ee90efa43d752: stopping tablet replica
I20260924 05:02:44.514132 18969 raft_consensus.cc:2229] T f20fe4258e62426c8d948cbd456f4542 P 7c775be5d37b4e1bb83ee90efa43d752 [term 1 LEADER]: Raft consensus shutting down.
I20260924 05:02:44.514216 18969 raft_consensus.cc:2258] T f20fe4258e62426c8d948cbd456f4542 P 7c775be5d37b4e1bb83ee90efa43d752 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:02:44.514537 18969 ts_tablet_manager.cc:1905] T f20fe4258e62426c8d948cbd456f4542 P 7c775be5d37b4e1bb83ee90efa43d752: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:02:44.515662 18969 ts_tablet_manager.cc:1918] T f20fe4258e62426c8d948cbd456f4542 P 7c775be5d37b4e1bb83ee90efa43d752: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 05:02:44.515717 18969 log.cc:1196] T f20fe4258e62426c8d948cbd456f4542 P 7c775be5d37b4e1bb83ee90efa43d752: Deleting WAL directory at /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790225929848825-392-0/minicluster-data/ts-2/wal/wals/f20fe4258e62426c8d948cbd456f4542
I20260924 05:02:44.515919 18969 ts_tablet_manager.cc:1939] T f20fe4258e62426c8d948cbd456f4542 P 7c775be5d37b4e1bb83ee90efa43d752: Deleting consensus metadata
I20260924 05:02:44.516225 18457 catalog_manager.cc:4841] TS 7c775be5d37b4e1bb83ee90efa43d752 (127.0.98.3:39699): tablet f20fe4258e62426c8d948cbd456f4542 (table test-table [id=4adceb1421de4654bc40aec054b53f67]) successfully deleted
W20260924 05:02:44.516734 18458 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38253 (error 108)
W20260924 05:02:44.518105 18939 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:02:44.517033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38253 (local address 127.0.0.1:49544)
0924 05:02:44.517080 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:44.517090 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:02:44.517124 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:02:44.518070 (+ 946us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38253: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":31}
W20260924 05:02:44.518215 18458 catalog_manager.cc:4584] TS fa0cc4f0f5064330b92de104a5a527af (127.0.98.1:38253): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f20fe4258e62426c8d948cbd456f4542: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38253: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 05:02:44.518482 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 18667
I20260924 05:02:44.523646 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 18801
I20260924 05:02:44.528002 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 18439
2026-09-24T05:02:44Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (1710 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-24T05:02:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:02:44Z Disabled control of system clock
I20260924 05:02:44.552814 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225929848825-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225929848825-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225929848825-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36695
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:45899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36695
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 05:02:44.631006 18974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:02:44.631208 18974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:02:44.631228 18974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:02:44.632308 18974 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:02:44.632341 18974 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:02:44.632352 18974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:02:44.632364 18974 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:02:44.632373 18974 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:02:44.633656 18974 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225929848825-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225929848825-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36695
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36695
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18974
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225929848825-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:02:44.633849 18974 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:02:44.634078 18974 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:02:44.638521 18982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:02:44.638593 18979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:02:44.638586 18980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:02:44.638716 18974 server_base.cc:1034] running on GCE node
I20260924 05:02:44.638959 18974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:02:44.639207 18974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:02:44.640355 18974 hybrid_clock.cc:648] HybridClock initialized: now 1790226164640343 us; error 28 us; skew 500 ppm
I20260924 05:02:44.641566 18974 webserver.cc:460] Webserver started at http://127.0.98.62:39127/ using document root <none> and password file <none>
I20260924 05:02:44.641791 18974 fs_manager.cc:359] Metadata directory not provided
I20260924 05:02:44.641840 18974 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:02:44.641960 18974 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:02:44.642946 18974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225929848825-392-0/minicluster-data/master-0/data/instance:
uuid: "204e19aaf6854ea896e91deb5d0a2cba"
format_stamp: "Formatted at 2026-09-24 05:02:44 on dist-test-slave-70st"
server_key: "b9d0aeefaa935224cad66cc6b7e36503"
server_key_iv: "748f1c6913a2ed7dd226ba8869528b6d"
server_key_version: "encryptionkey@0"
I20260924 05:02:44.643254 18974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225929848825-392-0/minicluster-data/master-0/wal/instance:
uuid: "204e19aaf6854ea896e91deb5d0a2cba"
format_stamp: "Formatted at 2026-09-24 05:02:44 on dist-test-slave-70st"
server_key: "b9d0aeefaa935224cad66cc6b7e36503"
server_key_iv: "748f1c6913a2ed7dd226ba8869528b6d"
server_key_version: "encryptionkey@0"
I20260924 05:02:44.644410 18974 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 05:02:44.645088 18988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:02:44.645260 18974 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:02:44.645335 18974 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225929848825-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225929848825-392-0/minicluster-data/master-0/wal
uuid: "204e19aaf6854ea896e91deb5d0a2cba"
format_stamp: "Formatted at 2026-09-24 05:02:44 on dist-test-slave-70st"
server_key: "b9d0aeefaa935224cad66cc6b7e36503"
server_key_iv: "748f1c6913a2ed7dd226ba8869528b6d"
server_key_version: "encryptionkey@0"
I20260924 05:02:44.645399 18974 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225929848825-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225929848825-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225929848825-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:02:44.669709 18974 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:02:44.670022 18974 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:02:44.670157 18974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:02:44.673560 18974 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36695
I20260924 05:02:44.673599 19040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36695 every 8 connection(s)
I20260924 05:02:44.673898 18974 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:02:44.674778 19041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:02:44.676721 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 18974
I20260924 05:02:44.676818 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225929848825-392-0/minicluster-data/master-0/wal/instance
I20260924 05:02:44.676957 19041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 204e19aaf6854ea896e91deb5d0a2cba: Bootstrap starting.
I20260924 05:02:44.676999 392 external_mini_cluster.cc:1428] Setting key 93fa84c580b9780ee0fc46ec9dc94f29
I20260924 05:02:44.677624 19041 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 204e19aaf6854ea896e91deb5d0a2cba: Neither blocks nor log segments found. Creating new log.
I20260924 05:02:44.677932 19041 log.cc:824] T 00000000000000000000000000000000 P 204e19aaf6854ea896e91deb5d0a2cba: Log is configured to *not* fsync() on all Append() calls
I20260924 05:02:44.678643 19041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 204e19aaf6854ea896e91deb5d0a2cba: No bootstrap required, opened a new log
I20260924 05:02:44.680730 19041 raft_consensus.cc:348] T 00000000000000000000000000000000 P 204e19aaf6854ea896e91deb5d0a2cba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "204e19aaf6854ea896e91deb5d0a2cba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36695 } }
I20260924 05:02:44.680892 19041 raft_consensus.cc:374] T 00000000000000000000000000000000 P 204e19aaf6854ea896e91deb5d0a2cba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:02:44.680923 19041 raft_consensus.cc:729] T 00000000000000000000000000000000 P 204e19aaf6854ea896e91deb5d0a2cba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 204e19aaf6854ea896e91deb5d0a2cba, State: Initialized, Role: FOLLOWER
I20260924 05:02:44.681039 19041 consensus_queue.cc:260] T 00000000000000000000000000000000 P 204e19aaf6854ea896e91deb5d0a2cba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "204e19aaf6854ea896e91deb5d0a2cba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36695 } }
I20260924 05:02:44.681106 19041 raft_consensus.cc:388] T 00000000000000000000000000000000 P 204e19aaf6854ea896e91deb5d0a2cba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:02:44.681139 19041 raft_consensus.cc:482] T 00000000000000000000000000000000 P 204e19aaf6854ea896e91deb5d0a2cba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:02:44.681180 19041 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 204e19aaf6854ea896e91deb5d0a2cba [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:02:44.681870 19041 raft_consensus.cc:504] T 00000000000000000000000000000000 P 204e19aaf6854ea896e91deb5d0a2cba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "204e19aaf6854ea896e91deb5d0a2cba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36695 } }
I20260924 05:02:44.681979 19041 leader_election.cc:302] T 00000000000000000000000000000000 P 204e19aaf6854ea896e91deb5d0a2cba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 204e19aaf6854ea896e91deb5d0a2cba; no voters:
I20260924 05:02:44.682147 19041 leader_election.cc:288] T 00000000000000000000000000000000 P 204e19aaf6854ea896e91deb5d0a2cba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:02:44.682188 19046 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 204e19aaf6854ea896e91deb5d0a2cba [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:02:44.682318 19046 raft_consensus.cc:686] T 00000000000000000000000000000000 P 204e19aaf6854ea896e91deb5d0a2cba [term 1 LEADER]: Becoming Leader. State: Replica: 204e19aaf6854ea896e91deb5d0a2cba, State: Running, Role: LEADER
I20260924 05:02:44.682413 19041 sys_catalog.cc:564] T 00000000000000000000000000000000 P 204e19aaf6854ea896e91deb5d0a2cba [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:02:44.682402 19046 consensus_queue.cc:237] T 00000000000000000000000000000000 P 204e19aaf6854ea896e91deb5d0a2cba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "204e19aaf6854ea896e91deb5d0a2cba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36695 } }
I20260924 05:02:44.682726 19046 sys_catalog.cc:455] T 00000000000000000000000000000000 P 204e19aaf6854ea896e91deb5d0a2cba [sys.catalog]: SysCatalogTable state changed. Reason: New leader 204e19aaf6854ea896e91deb5d0a2cba. Latest consensus state: current_term: 1 leader_uuid: "204e19aaf6854ea896e91deb5d0a2cba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "204e19aaf6854ea896e91deb5d0a2cba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36695 } } }
I20260924 05:02:44.682808 19046 sys_catalog.cc:458] T 00000000000000000000000000000000 P 204e19aaf6854ea896e91deb5d0a2cba [sys.catalog]: This master's current role is: LEADER
I20260924 05:02:44.682736 19048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 204e19aaf6854ea896e91deb5d0a2cba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "204e19aaf6854ea896e91deb5d0a2cba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "204e19aaf6854ea896e91deb5d0a2cba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36695 } } }
I20260924 05:02:44.682952 19048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 204e19aaf6854ea896e91deb5d0a2cba [sys.catalog]: This master's current role is: LEADER
I20260924 05:02:44.682983 19044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:44.677483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53140 (local address 127.0.98.62:36695)
0924 05:02:44.677692 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0924 05:02:44.677701 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 05:02:44.677736 (+ 35us) server_negotiation.cc:407] Connection header received
0924 05:02:44.678263 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:02:44.678270 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:02:44.678332 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:02:44.678479 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:02:44.678653 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:44.680013 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:44.680767 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:44.681348 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:44.681492 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:44.682069 (+ 577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:02:44.682073 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:02:44.682074 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:02:44.682272 (+ 198us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:02:44.682313 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:02:44.682329 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:02:44.682638 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0924 05:02:44.682672 (+ 34us) server_negotiation.cc:299] Negotiation successful
0924 05:02:44.682747 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":74,"threads_started":1}
I20260924 05:02:44.683007 19051 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:02:44.684199 19051 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:02:44.685549 19051 catalog_manager.cc:1347] Generated new cluster ID: 595ab1e6d4a345c98862d9d3214e827e
I20260924 05:02:44.685601 19051 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:02:44.700614 19051 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:02:44.701305 19051 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:02:44.705751 19051 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 204e19aaf6854ea896e91deb5d0a2cba: Generated new TSK 0
I20260924 05:02:44.705921 19051 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 05:02:44.715605 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225929848825-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225929848825-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225929848825-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225929848825-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36695
--builtin_ntp_servers=127.0.98.20:45899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 05:02:44.794023 19065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:02:44.794227 19065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:02:44.794250 19065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:02:44.795476 19065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:02:44.795521 19065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:02:44.795547 19065 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 05:02:44.796911 19065 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225929848825-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225929848825-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225929848825-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36695
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19065
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225929848825-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:02:44.797134 19065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:02:44.797444 19065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:02:44.802153 19070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:02:44.802192 19071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:02:44.802207 19073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:02:44.802654 19065 server_base.cc:1034] running on GCE node
I20260924 05:02:44.802855 19065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:02:44.803079 19065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:02:44.804215 19065 hybrid_clock.cc:648] HybridClock initialized: now 1790226164804200 us; error 29 us; skew 500 ppm
I20260924 05:02:44.805456 19065 webserver.cc:460] Webserver started at http://127.0.98.1:44801/ using document root <none> and password file <none>
I20260924 05:02:44.805672 19065 fs_manager.cc:359] Metadata directory not provided
I20260924 05:02:44.805718 19065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:02:44.805840 19065 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:02:44.806860 19065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225929848825-392-0/minicluster-data/ts-0/data/instance:
uuid: "d9b0d2d018c64b12b010d6f191631416"
format_stamp: "Formatted at 2026-09-24 05:02:44 on dist-test-slave-70st"
server_key: "cb6740a80449f1ee888ce39a43ba5a7b"
server_key_iv: "5a4f5c2f86a95b888ec8bc9e676c5e0b"
server_key_version: "encryptionkey@0"
I20260924 05:02:44.807186 19065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225929848825-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d9b0d2d018c64b12b010d6f191631416"
format_stamp: "Formatted at 2026-09-24 05:02:44 on dist-test-slave-70st"
server_key: "cb6740a80449f1ee888ce39a43ba5a7b"
server_key_iv: "5a4f5c2f86a95b888ec8bc9e676c5e0b"
server_key_version: "encryptionkey@0"
I20260924 05:02:44.808415 19065 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 05:02:44.809119 19079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:02:44.809267 19065 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:02:44.809345 19065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225929848825-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225929848825-392-0/minicluster-data/ts-0/wal
uuid: "d9b0d2d018c64b12b010d6f191631416"
format_stamp: "Formatted at 2026-09-24 05:02:44 on dist-test-slave-70st"
server_key: "cb6740a80449f1ee888ce39a43ba5a7b"
server_key_iv: "5a4f5c2f86a95b888ec8bc9e676c5e0b"
server_key_version: "encryptionkey@0"
I20260924 05:02:44.809470 19065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225929848825-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225929848825-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225929848825-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:02:44.832126 19065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:02:44.832468 19065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:02:44.832599 19065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:02:44.832814 19065 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:02:44.833143 19065 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:02:44.833176 19065 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:02:44.833206 19065 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:02:44.833228 19065 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:02:44.838603 19065 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46273
I20260924 05:02:44.838651 19192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46273 every 8 connection(s)
I20260924 05:02:44.838953 19065 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225929848825-392-0/minicluster-data/ts-0/data/info.pb
I20260924 05:02:44.840077 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 19065
I20260924 05:02:44.840183 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225929848825-392-0/minicluster-data/ts-0/wal/instance
I20260924 05:02:44.840382 392 external_mini_cluster.cc:1428] Setting key e14d6a822e63dbc4a2a6c9b069907051
I20260924 05:02:44.841367 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225929848825-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225929848825-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225929848825-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225929848825-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36695
--builtin_ntp_servers=127.0.98.20:45899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 05:02:44.844585 19044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:44.839614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40097 (local address 127.0.98.62:36695)
0924 05:02:44.839684 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 05:02:44.839686 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:02:44.839926 (+ 240us) server_negotiation.cc:407] Connection header received
0924 05:02:44.840618 (+ 692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:02:44.840619 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:02:44.840631 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:02:44.840681 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:02:44.841535 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:44.842214 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:44.843185 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:44.843478 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:44.843544 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:44.844316 (+ 772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:02:44.844317 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:02:44.844317 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:02:44.844322 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:02:44.844340 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:02:44.844344 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:02:44.844395 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 05:02:44.844471 (+ 76us) server_negotiation.cc:299] Negotiation successful
0924 05:02:44.844502 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 05:02:44.844735 19195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:44.839585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36695 (local address 127.0.98.1:40097)
0924 05:02:44.839756 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:44.839806 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 05:02:44.840417 (+ 611us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:02:44.840769 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:02:44.840774 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:02:44.840784 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:02:44.841384 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:44.841390 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:44.842292 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:44.842293 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:44.843097 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:44.843100 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:44.843550 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:44.843551 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:44.843669 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:44.844171 (+ 502us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 05:02:44.844172 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:02:44.844201 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 05:02:44.844225 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:02:44.844402 (+ 177us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:02:44.844406 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:02:44.844408 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 05:02:44.844460 (+ 52us) client_negotiation.cc:240] Negotiation successful
0924 05:02:44.844544 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":72,"threads_started":1}
I20260924 05:02:44.845227 19193 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36695
I20260924 05:02:44.845314 19193 heartbeater.cc:464] Registering TS with master...
I20260924 05:02:44.845536 19193 heartbeater.cc:511] Master 127.0.98.62:36695 requested a full tablet report, sending...
I20260924 05:02:44.846012 19005 ts_manager.cc:194] Registered new tserver with Master: d9b0d2d018c64b12b010d6f191631416 (127.0.98.1:46273)
I20260924 05:02:44.846642 19005 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:40097
W20260924 05:02:44.919577 19196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:02:44.919773 19196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:02:44.919793 19196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:02:44.920902 19196 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:02:44.920939 19196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:02:44.920965 19196 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 05:02:44.922294 19196 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225929848825-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225929848825-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225929848825-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36695
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19196
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225929848825-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:02:44.922509 19196 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:02:44.922819 19196 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:02:44.927615 19204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:02:44.927628 19202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:02:44.927716 19201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:02:44.927826 19196 server_base.cc:1034] running on GCE node
I20260924 05:02:44.928001 19196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:02:44.928179 19196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:02:44.931392 19196 hybrid_clock.cc:648] HybridClock initialized: now 1790226164930910 us; error 492 us; skew 500 ppm
I20260924 05:02:44.932583 19196 webserver.cc:460] Webserver started at http://127.0.98.2:36853/ using document root <none> and password file <none>
I20260924 05:02:44.932793 19196 fs_manager.cc:359] Metadata directory not provided
I20260924 05:02:44.932833 19196 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:02:44.932922 19196 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:02:44.933998 19196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225929848825-392-0/minicluster-data/ts-1/data/instance:
uuid: "a018f832db2a415989e154108749fd85"
format_stamp: "Formatted at 2026-09-24 05:02:44 on dist-test-slave-70st"
server_key: "c3911e247f8905cc832e5eeadc596db8"
server_key_iv: "b4a3acc4b5abfb3daa111edc46496ce0"
server_key_version: "encryptionkey@0"
I20260924 05:02:44.934300 19196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225929848825-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a018f832db2a415989e154108749fd85"
format_stamp: "Formatted at 2026-09-24 05:02:44 on dist-test-slave-70st"
server_key: "c3911e247f8905cc832e5eeadc596db8"
server_key_iv: "b4a3acc4b5abfb3daa111edc46496ce0"
server_key_version: "encryptionkey@0"
I20260924 05:02:44.935439 19196 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 05:02:44.936077 19210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:02:44.936219 19196 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:02:44.936297 19196 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225929848825-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225929848825-392-0/minicluster-data/ts-1/wal
uuid: "a018f832db2a415989e154108749fd85"
format_stamp: "Formatted at 2026-09-24 05:02:44 on dist-test-slave-70st"
server_key: "c3911e247f8905cc832e5eeadc596db8"
server_key_iv: "b4a3acc4b5abfb3daa111edc46496ce0"
server_key_version: "encryptionkey@0"
I20260924 05:02:44.936358 19196 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225929848825-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225929848825-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225929848825-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:02:44.954799 19196 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:02:44.955106 19196 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:02:44.955231 19196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:02:44.955443 19196 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:02:44.955801 19196 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:02:44.955838 19196 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:02:44.955868 19196 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:02:44.955892 19196 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:02:44.961344 19196 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42175
I20260924 05:02:44.961400 19323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42175 every 8 connection(s)
I20260924 05:02:44.961766 19196 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225929848825-392-0/minicluster-data/ts-1/data/info.pb
I20260924 05:02:44.965975 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 19196
I20260924 05:02:44.966066 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225929848825-392-0/minicluster-data/ts-1/wal/instance
I20260924 05:02:44.966223 392 external_mini_cluster.cc:1428] Setting key e9bb340e55a32fe6a90474c0f6734792
I20260924 05:02:44.967013 19044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:44.962491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54471 (local address 127.0.98.62:36695)
0924 05:02:44.962569 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0924 05:02:44.962570 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:02:44.962806 (+ 236us) server_negotiation.cc:407] Connection header received
0924 05:02:44.963270 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:02:44.963271 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:02:44.963284 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:02:44.963339 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:02:44.964013 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:44.964516 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:44.965521 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:44.965714 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:44.965797 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:44.966651 (+ 854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:02:44.966653 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:02:44.966653 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:02:44.966657 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:02:44.966674 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:02:44.966678 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:02:44.966732 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 05:02:44.966907 (+ 175us) server_negotiation.cc:299] Negotiation successful
0924 05:02:44.966936 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 05:02:44.967123 19326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:44.962413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36695 (local address 127.0.98.2:54471)
0924 05:02:44.962657 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:44.962693 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 05:02:44.963190 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:02:44.963410 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:02:44.963413 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:02:44.963422 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:02:44.963934 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:44.963939 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:44.964592 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:44.964593 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:44.965409 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:44.965441 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:44.965799 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:44.965800 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:44.965920 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:44.966535 (+ 615us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 05:02:44.966535 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:02:44.966565 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 05:02:44.966589 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:02:44.966736 (+ 147us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:02:44.966739 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:02:44.966742 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:02:44.966818 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 05:02:44.966920 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":86,"threads_started":1}
I20260924 05:02:44.967181 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225929848825-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225929848825-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225929848825-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225929848825-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36695
--builtin_ntp_servers=127.0.98.20:45899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 05:02:44.967576 19324 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36695
I20260924 05:02:44.967661 19324 heartbeater.cc:464] Registering TS with master...
I20260924 05:02:44.967831 19324 heartbeater.cc:511] Master 127.0.98.62:36695 requested a full tablet report, sending...
I20260924 05:02:44.968153 19005 ts_manager.cc:194] Registered new tserver with Master: a018f832db2a415989e154108749fd85 (127.0.98.2:42175)
I20260924 05:02:44.968498 19005 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:54471
W20260924 05:02:45.048394 19327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:02:45.048620 19327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:02:45.048641 19327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:02:45.049863 19327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:02:45.049904 19327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:02:45.049929 19327 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 05:02:45.051249 19327 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225929848825-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225929848825-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225929848825-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36695
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19327
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225929848825-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:02:45.051544 19327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:02:45.051820 19327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:02:45.056519 19332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:02:45.056541 19333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:02:45.056550 19335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:02:45.056607 19327 server_base.cc:1034] running on GCE node
I20260924 05:02:45.056905 19327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:02:45.057134 19327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:02:45.058305 19327 hybrid_clock.cc:648] HybridClock initialized: now 1790226165058287 us; error 28 us; skew 500 ppm
I20260924 05:02:45.059643 19327 webserver.cc:460] Webserver started at http://127.0.98.3:33403/ using document root <none> and password file <none>
I20260924 05:02:45.059849 19327 fs_manager.cc:359] Metadata directory not provided
I20260924 05:02:45.059916 19327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:02:45.060038 19327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:02:45.061030 19327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225929848825-392-0/minicluster-data/ts-2/data/instance:
uuid: "07d06eb973b6438188660241a586e97b"
format_stamp: "Formatted at 2026-09-24 05:02:45 on dist-test-slave-70st"
server_key: "c9b7662053ae0aebd27954211adec20a"
server_key_iv: "4f2b9fb88373108bc0bcb007e8d508fb"
server_key_version: "encryptionkey@0"
I20260924 05:02:45.061347 19327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225929848825-392-0/minicluster-data/ts-2/wal/instance:
uuid: "07d06eb973b6438188660241a586e97b"
format_stamp: "Formatted at 2026-09-24 05:02:45 on dist-test-slave-70st"
server_key: "c9b7662053ae0aebd27954211adec20a"
server_key_iv: "4f2b9fb88373108bc0bcb007e8d508fb"
server_key_version: "encryptionkey@0"
I20260924 05:02:45.062552 19327 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 05:02:45.063261 19341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:02:45.063416 19327 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 05:02:45.063495 19327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225929848825-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225929848825-392-0/minicluster-data/ts-2/wal
uuid: "07d06eb973b6438188660241a586e97b"
format_stamp: "Formatted at 2026-09-24 05:02:45 on dist-test-slave-70st"
server_key: "c9b7662053ae0aebd27954211adec20a"
server_key_iv: "4f2b9fb88373108bc0bcb007e8d508fb"
server_key_version: "encryptionkey@0"
I20260924 05:02:45.063553 19327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225929848825-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225929848825-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225929848825-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:02:45.077652 19327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:02:45.077937 19327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:02:45.078063 19327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:02:45.078286 19327 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:02:45.078614 19327 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:02:45.078646 19327 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:02:45.078673 19327 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:02:45.078691 19327 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:02:45.084105 19327 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41753
I20260924 05:02:45.084157 19454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41753 every 8 connection(s)
I20260924 05:02:45.084467 19327 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225929848825-392-0/minicluster-data/ts-2/data/info.pb
I20260924 05:02:45.089202 19044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:45.085132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56395 (local address 127.0.98.62:36695)
0924 05:02:45.085175 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 05:02:45.085177 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:02:45.085383 (+ 206us) server_negotiation.cc:407] Connection header received
0924 05:02:45.085905 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:02:45.085906 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:02:45.085916 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:02:45.085953 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:02:45.086585 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:45.087073 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:45.088026 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:45.088213 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:45.088278 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:45.089018 (+ 740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:02:45.089018 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:02:45.089019 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:02:45.089022 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:02:45.089033 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:02:45.089036 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:02:45.089068 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0924 05:02:45.089126 (+ 58us) server_negotiation.cc:299] Negotiation successful
0924 05:02:45.089144 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 05:02:45.089392 19457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:45.085121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36695 (local address 127.0.98.3:56395)
0924 05:02:45.085282 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:45.085314 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 05:02:45.085809 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:02:45.086009 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:02:45.086012 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:02:45.086020 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:02:45.086491 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:45.086495 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:45.087160 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:45.087160 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:45.087948 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:45.087950 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:45.088282 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:45.088283 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:45.088385 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:45.088902 (+ 517us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 05:02:45.088902 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:02:45.088930 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 05:02:45.088952 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:02:45.089074 (+ 122us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:02:45.089076 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:02:45.089078 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 05:02:45.089121 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 05:02:45.089193 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":80,"threads_started":1}
I20260924 05:02:45.089848 19455 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36695
I20260924 05:02:45.089932 19455 heartbeater.cc:464] Registering TS with master...
I20260924 05:02:45.090098 19455 heartbeater.cc:511] Master 127.0.98.62:36695 requested a full tablet report, sending...
I20260924 05:02:45.090438 19005 ts_manager.cc:194] Registered new tserver with Master: 07d06eb973b6438188660241a586e97b (127.0.98.3:41753)
I20260924 05:02:45.090768 19005 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:56395
I20260924 05:02:45.092044 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 19327
I20260924 05:02:45.092145 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225929848825-392-0/minicluster-data/ts-2/wal/instance
I20260924 05:02:45.092325 392 external_mini_cluster.cc:1428] Setting key e39d4c0a798420c1f8537e0b30f4e820
I20260924 05:02:45.093084 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 05:02:45.096973 19044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:45.094376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53154 (local address 127.0.98.62:36695)
0924 05:02:45.094430 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 05:02:45.094431 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:02:45.094507 (+ 76us) server_negotiation.cc:407] Connection header received
0924 05:02:45.094556 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:02:45.094557 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:02:45.094564 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:02:45.094599 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:02:45.094716 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:45.095176 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:45.095930 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:45.096104 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:45.096151 (+ 47us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:45.096763 (+ 612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:02:45.096764 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:02:45.096764 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:02:45.096768 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:02:45.096778 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:02:45.096781 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:02:45.096843 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 05:02:45.096913 (+ 70us) server_negotiation.cc:299] Negotiation successful
0924 05:02:45.096921 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 05:02:45.098315 19005 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53154:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 05:02:45.099834 19005 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 05:02:45.105861 19464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:45.101818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41753 (local address 127.0.0.1:53638)
0924 05:02:45.101983 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:45.102020 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 05:02:45.102084 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:02:45.102502 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:02:45.102505 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:02:45.102507 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:02:45.102599 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:45.102601 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:45.103906 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:45.103907 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:45.104757 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:45.104759 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:45.105546 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:45.105547 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:45.105622 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:45.105636 (+ 14us) client_negotiation.cc:769] Sending connection context
0924 05:02:45.105713 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 05:02:45.105764 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":19,"thread_start_us":57,"threads_started":1}
I20260924 05:02:45.105860 19466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:45.101982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53638 (local address 127.0.98.3:41753)
0924 05:02:45.102360 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0924 05:02:45.102364 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 05:02:45.102373 (+ 9us) server_negotiation.cc:407] Connection header received
0924 05:02:45.102403 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:02:45.102405 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:02:45.102420 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:02:45.102487 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:02:45.102687 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:45.103807 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:45.104844 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:45.105478 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:45.105547 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:45.105636 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 05:02:45.105729 (+ 93us) server_negotiation.cc:299] Negotiation successful
0924 05:02:45.105768 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":80,"threads_started":1}
I20260924 05:02:45.106211 19463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:45.101837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42175 (local address 127.0.0.1:59092)
0924 05:02:45.102584 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:45.102593 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:02:45.102639 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:02:45.102882 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:02:45.102883 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:02:45.102885 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:02:45.102925 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:45.102926 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:45.103736 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:45.103737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:45.104992 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:45.104995 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:45.105991 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:45.105992 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:45.106054 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:45.106058 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 05:02:45.106109 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 05:02:45.106145 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":735,"mutex_wait_us":28,"thread_start_us":87,"threads_started":1}
I20260924 05:02:45.106263 19467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:45.102044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59092 (local address 127.0.98.2:42175)
0924 05:02:45.102231 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 05:02:45.102236 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 05:02:45.102698 (+ 462us) server_negotiation.cc:407] Connection header received
0924 05:02:45.102741 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:02:45.102743 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:02:45.102758 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:02:45.102821 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:02:45.102992 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:45.103663 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:45.105082 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:45.105808 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:45.105871 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:45.105953 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 05:02:45.106141 (+ 188us) server_negotiation.cc:299] Negotiation successful
0924 05:02:45.106183 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":84,"threads_started":1}
I20260924 05:02:45.106761 19389 tablet_service.cc:1467] Processing CreateTablet for tablet 4bcb40cc4e3146ac9f3d3574a2c7a531 (DEFAULT_TABLE table=test-table [id=2c7fce36d4d243f1a5f6d113374ab80b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:02:45.107086 19389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bcb40cc4e3146ac9f3d3574a2c7a531. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:02:45.109530 19469 tablet_bootstrap.cc:492] T 4bcb40cc4e3146ac9f3d3574a2c7a531 P 07d06eb973b6438188660241a586e97b: Bootstrap starting.
I20260924 05:02:45.109840 19258 tablet_service.cc:1467] Processing CreateTablet for tablet 4bcb40cc4e3146ac9f3d3574a2c7a531 (DEFAULT_TABLE table=test-table [id=2c7fce36d4d243f1a5f6d113374ab80b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:02:45.110146 19258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bcb40cc4e3146ac9f3d3574a2c7a531. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:02:45.110193 19469 tablet_bootstrap.cc:654] T 4bcb40cc4e3146ac9f3d3574a2c7a531 P 07d06eb973b6438188660241a586e97b: Neither blocks nor log segments found. Creating new log.
I20260924 05:02:45.110456 19469 log.cc:824] T 4bcb40cc4e3146ac9f3d3574a2c7a531 P 07d06eb973b6438188660241a586e97b: Log is configured to *not* fsync() on all Append() calls
I20260924 05:02:45.111047 19469 tablet_bootstrap.cc:492] T 4bcb40cc4e3146ac9f3d3574a2c7a531 P 07d06eb973b6438188660241a586e97b: No bootstrap required, opened a new log
I20260924 05:02:45.111099 19469 ts_tablet_manager.cc:1397] T 4bcb40cc4e3146ac9f3d3574a2c7a531 P 07d06eb973b6438188660241a586e97b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 05:02:45.112600 19471 tablet_bootstrap.cc:492] T 4bcb40cc4e3146ac9f3d3574a2c7a531 P a018f832db2a415989e154108749fd85: Bootstrap starting.
I20260924 05:02:45.112757 19465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:45.101829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46273 (local address 127.0.0.1:41852)
0924 05:02:45.102065 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:45.102075 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:02:45.102126 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:02:45.102501 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:02:45.102505 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:02:45.102509 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:02:45.102599 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:45.102601 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:45.103887 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:45.103888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:45.105006 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:45.105007 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:45.106367 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:45.106368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:45.112580 (+ 6212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:45.112587 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 05:02:45.112637 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 05:02:45.112683 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"mutex_wait_us":32,"thread_start_us":83,"threads_started":1}
I20260924 05:02:45.113040 19468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:45.102154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41852 (local address 127.0.98.1:46273)
0924 05:02:45.102316 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 05:02:45.102321 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 05:02:45.102331 (+ 10us) server_negotiation.cc:407] Connection header received
0924 05:02:45.102373 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:02:45.102376 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:02:45.102391 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:02:45.102476 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:02:45.102687 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:45.103813 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:45.105485 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:45.106177 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:45.106239 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:45.106331 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 05:02:45.112912 (+ 6581us) server_negotiation.cc:299] Negotiation successful
0924 05:02:45.112965 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":84,"threads_started":1}
I20260924 05:02:45.113111 19469 raft_consensus.cc:348] T 4bcb40cc4e3146ac9f3d3574a2c7a531 P 07d06eb973b6438188660241a586e97b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9b0d2d018c64b12b010d6f191631416" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46273 } } peers { permanent_uuid: "07d06eb973b6438188660241a586e97b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41753 } } peers { permanent_uuid: "a018f832db2a415989e154108749fd85" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42175 } }
I20260924 05:02:45.113272 19471 tablet_bootstrap.cc:654] T 4bcb40cc4e3146ac9f3d3574a2c7a531 P a018f832db2a415989e154108749fd85: Neither blocks nor log segments found. Creating new log.
I20260924 05:02:45.113274 19469 raft_consensus.cc:374] T 4bcb40cc4e3146ac9f3d3574a2c7a531 P 07d06eb973b6438188660241a586e97b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:02:45.113309 19469 raft_consensus.cc:729] T 4bcb40cc4e3146ac9f3d3574a2c7a531 P 07d06eb973b6438188660241a586e97b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07d06eb973b6438188660241a586e97b, State: Initialized, Role: FOLLOWER
I20260924 05:02:45.113482 19469 consensus_queue.cc:260] T 4bcb40cc4e3146ac9f3d3574a2c7a531 P 07d06eb973b6438188660241a586e97b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9b0d2d018c64b12b010d6f191631416" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46273 } } peers { permanent_uuid: "07d06eb973b6438188660241a586e97b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41753 } } peers { permanent_uuid: "a018f832db2a415989e154108749fd85" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42175 } }
I20260924 05:02:45.113581 19471 log.cc:824] T 4bcb40cc4e3146ac9f3d3574a2c7a531 P a018f832db2a415989e154108749fd85: Log is configured to *not* fsync() on all Append() calls
I20260924 05:02:45.113736 19469 ts_tablet_manager.cc:1428] T 4bcb40cc4e3146ac9f3d3574a2c7a531 P 07d06eb973b6438188660241a586e97b: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260924 05:02:45.113683 19127 tablet_service.cc:1467] Processing CreateTablet for tablet 4bcb40cc4e3146ac9f3d3574a2c7a531 (DEFAULT_TABLE table=test-table [id=2c7fce36d4d243f1a5f6d113374ab80b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:02:45.114015 19127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bcb40cc4e3146ac9f3d3574a2c7a531. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:02:45.114288 19471 tablet_bootstrap.cc:492] T 4bcb40cc4e3146ac9f3d3574a2c7a531 P a018f832db2a415989e154108749fd85: No bootstrap required, opened a new log
I20260924 05:02:45.114347 19471 ts_tablet_manager.cc:1397] T 4bcb40cc4e3146ac9f3d3574a2c7a531 P a018f832db2a415989e154108749fd85: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 05:02:45.116089 19471 raft_consensus.cc:348] T 4bcb40cc4e3146ac9f3d3574a2c7a531 P a018f832db2a415989e154108749fd85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9b0d2d018c64b12b010d6f191631416" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46273 } } peers { permanent_uuid: "07d06eb973b6438188660241a586e97b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41753 } } peers { permanent_uuid: "a018f832db2a415989e154108749fd85" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42175 } }
I20260924 05:02:45.116225 19471 raft_consensus.cc:374] T 4bcb40cc4e3146ac9f3d3574a2c7a531 P a018f832db2a415989e154108749fd85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:02:45.116259 19471 raft_consensus.cc:729] T 4bcb40cc4e3146ac9f3d3574a2c7a531 P a018f832db2a415989e154108749fd85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a018f832db2a415989e154108749fd85, State: Initialized, Role: FOLLOWER
I20260924 05:02:45.116294 19455 heartbeater.cc:503] Master 127.0.98.62:36695 was elected leader, sending a full tablet report...
I20260924 05:02:45.116317 19474 tablet_bootstrap.cc:492] T 4bcb40cc4e3146ac9f3d3574a2c7a531 P d9b0d2d018c64b12b010d6f191631416: Bootstrap starting.
I20260924 05:02:45.116374 19471 consensus_queue.cc:260] T 4bcb40cc4e3146ac9f3d3574a2c7a531 P a018f832db2a415989e154108749fd85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9b0d2d018c64b12b010d6f191631416" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46273 } } peers { permanent_uuid: "07d06eb973b6438188660241a586e97b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41753 } } peers { permanent_uuid: "a018f832db2a415989e154108749fd85" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42175 } }
I20260924 05:02:45.116632 19471 ts_tablet_manager.cc:1428] T 4bcb40cc4e3146ac9f3d3574a2c7a531 P a018f832db2a415989e154108749fd85: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260924 05:02:45.116680 19324 heartbeater.cc:503] Master 127.0.98.62:36695 was elected leader, sending a full tablet report...
I20260924 05:02:45.117370 19474 tablet_bootstrap.cc:654] T 4bcb40cc4e3146ac9f3d3574a2c7a531 P d9b0d2d018c64b12b010d6f191631416: Neither blocks nor log segments found. Creating new log.
I20260924 05:02:45.117652 19474 log.cc:824] T 4bcb40cc4e3146ac9f3d3574a2c7a531 P d9b0d2d018c64b12b010d6f191631416: Log is configured to *not* fsync() on all Append() calls
I20260924 05:02:45.118249 19474 tablet_bootstrap.cc:492] T 4bcb40cc4e3146ac9f3d3574a2c7a531 P d9b0d2d018c64b12b010d6f191631416: No bootstrap required, opened a new log
I20260924 05:02:45.118307 19474 ts_tablet_manager.cc:1397] T 4bcb40cc4e3146ac9f3d3574a2c7a531 P d9b0d2d018c64b12b010d6f191631416: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 05:02:45.119968 19474 raft_consensus.cc:348] T 4bcb40cc4e3146ac9f3d3574a2c7a531 P d9b0d2d018c64b12b010d6f191631416 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9b0d2d018c64b12b010d6f191631416" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46273 } } peers { permanent_uuid: "07d06eb973b6438188660241a586e97b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41753 } } peers { permanent_uuid: "a018f832db2a415989e154108749fd85" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42175 } }
I20260924 05:02:45.120087 19474 raft_consensus.cc:374] T 4bcb40cc4e3146ac9f3d3574a2c7a531 P d9b0d2d018c64b12b010d6f191631416 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:02:45.120118 19474 raft_consensus.cc:729] T 4bcb40cc4e3146ac9f3d3574a2c7a531 P d9b0d2d018c64b12b010d6f191631416 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9b0d2d018c64b12b010d6f191631416, State: Initialized, Role: FOLLOWER
I20260924 05:02:45.120223 19474 consensus_queue.cc:260] T 4bcb40cc4e3146ac9f3d3574a2c7a531 P d9b0d2d018c64b12b010d6f191631416 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9b0d2d018c64b12b010d6f191631416" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46273 } } peers { permanent_uuid: "07d06eb973b6438188660241a586e97b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41753 } } peers { permanent_uuid: "a018f832db2a415989e154108749fd85" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42175 } }
I20260924 05:02:45.120453 19474 ts_tablet_manager.cc:1428] T 4bcb40cc4e3146ac9f3d3574a2c7a531 P d9b0d2d018c64b12b010d6f191631416: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 05:02:45.120532 19193 heartbeater.cc:503] Master 127.0.98.62:36695 was elected leader, sending a full tablet report...
I20260924 05:02:45.175594 19472 raft_consensus.cc:482] T 4bcb40cc4e3146ac9f3d3574a2c7a531 P 07d06eb973b6438188660241a586e97b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 05:02:45.175724 19472 raft_consensus.cc:504] T 4bcb40cc4e3146ac9f3d3574a2c7a531 P 07d06eb973b6438188660241a586e97b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9b0d2d018c64b12b010d6f191631416" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46273 } } peers { permanent_uuid: "07d06eb973b6438188660241a586e97b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41753 } } peers { permanent_uuid: "a018f832db2a415989e154108749fd85" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42175 } }
I20260924 05:02:45.176090 19472 leader_election.cc:288] T 4bcb40cc4e3146ac9f3d3574a2c7a531 P 07d06eb973b6438188660241a586e97b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9b0d2d018c64b12b010d6f191631416 (127.0.98.1:46273), a018f832db2a415989e154108749fd85 (127.0.98.2:42175)
I20260924 05:02:45.178807 19467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:45.176345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44315 (local address 127.0.98.2:42175)
0924 05:02:45.176391 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 05:02:45.176393 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:02:45.176404 (+ 11us) server_negotiation.cc:407] Connection header received
0924 05:02:45.176472 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:02:45.176473 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:02:45.176483 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:02:45.176541 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:02:45.176686 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:45.177163 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:45.178130 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:45.178504 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:45.178562 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:45.178574 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0924 05:02:45.178702 (+ 128us) server_negotiation.cc:299] Negotiation successful
0924 05:02:45.178735 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 05:02:45.178807 19457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:45.176276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42175 (local address 127.0.98.3:44315)
0924 05:02:45.176357 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:45.176369 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 05:02:45.176412 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:02:45.176557 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:02:45.176558 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:02:45.176560 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:02:45.176626 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:45.176628 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:45.177237 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:45.177238 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:45.178042 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:45.178044 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:45.178575 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:45.178576 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:45.178642 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:45.178645 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:02:45.178688 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 05:02:45.178729 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260924 05:02:45.178927 19468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:45.176360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34779 (local address 127.0.98.1:46273)
0924 05:02:45.176411 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 05:02:45.176412 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:02:45.176580 (+ 168us) server_negotiation.cc:407] Connection header received
0924 05:02:45.176607 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:02:45.176608 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:02:45.176619 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:02:45.176660 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:02:45.176797 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:45.177250 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:45.178202 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:45.178608 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:45.178670 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:45.178679 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 05:02:45.178820 (+ 141us) server_negotiation.cc:299] Negotiation successful
0924 05:02:45.178865 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 05:02:45.178946 19478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:45.176300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46273 (local address 127.0.98.3:34779)
0924 05:02:45.176506 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:45.176512 (+ 6us) client_negotiation.cc:174] Beginning negotiation
0924 05:02:45.176548 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:02:45.176670 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:02:45.176671 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:02:45.176672 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:02:45.176737 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:45.176739 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:45.177322 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:45.177323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:45.178113 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:45.178115 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:45.178680 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:45.178681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:45.178759 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:45.178762 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:02:45.178806 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 05:02:45.178841 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"mutex_wait_us":3,"thread_start_us":135,"threads_started":1}
I20260924 05:02:45.179183 19278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bcb40cc4e3146ac9f3d3574a2c7a531" candidate_uuid: "07d06eb973b6438188660241a586e97b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a018f832db2a415989e154108749fd85" is_pre_election: true
I20260924 05:02:45.179358 19278 raft_consensus.cc:2454] T 4bcb40cc4e3146ac9f3d3574a2c7a531 P a018f832db2a415989e154108749fd85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07d06eb973b6438188660241a586e97b in term 0.
I20260924 05:02:45.179311 19147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bcb40cc4e3146ac9f3d3574a2c7a531" candidate_uuid: "07d06eb973b6438188660241a586e97b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9b0d2d018c64b12b010d6f191631416" is_pre_election: true
I20260924 05:02:45.179458 19147 raft_consensus.cc:2454] T 4bcb40cc4e3146ac9f3d3574a2c7a531 P d9b0d2d018c64b12b010d6f191631416 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07d06eb973b6438188660241a586e97b in term 0.
I20260924 05:02:45.179582 19343 leader_election.cc:302] T 4bcb40cc4e3146ac9f3d3574a2c7a531 P 07d06eb973b6438188660241a586e97b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07d06eb973b6438188660241a586e97b, a018f832db2a415989e154108749fd85; no voters:
I20260924 05:02:45.179711 19472 raft_consensus.cc:2781] T 4bcb40cc4e3146ac9f3d3574a2c7a531 P 07d06eb973b6438188660241a586e97b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 05:02:45.179781 19472 raft_consensus.cc:482] T 4bcb40cc4e3146ac9f3d3574a2c7a531 P 07d06eb973b6438188660241a586e97b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 05:02:45.179797 19472 raft_consensus.cc:3037] T 4bcb40cc4e3146ac9f3d3574a2c7a531 P 07d06eb973b6438188660241a586e97b [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:02:45.180469 19472 raft_consensus.cc:504] T 4bcb40cc4e3146ac9f3d3574a2c7a531 P 07d06eb973b6438188660241a586e97b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9b0d2d018c64b12b010d6f191631416" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46273 } } peers { permanent_uuid: "07d06eb973b6438188660241a586e97b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41753 } } peers { permanent_uuid: "a018f832db2a415989e154108749fd85" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42175 } }
I20260924 05:02:45.180589 19472 leader_election.cc:288] T 4bcb40cc4e3146ac9f3d3574a2c7a531 P 07d06eb973b6438188660241a586e97b [CANDIDATE]: Term 1 election: Requested vote from peers d9b0d2d018c64b12b010d6f191631416 (127.0.98.1:46273), a018f832db2a415989e154108749fd85 (127.0.98.2:42175)
I20260924 05:02:45.180716 19147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bcb40cc4e3146ac9f3d3574a2c7a531" candidate_uuid: "07d06eb973b6438188660241a586e97b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9b0d2d018c64b12b010d6f191631416"
I20260924 05:02:45.180734 19278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bcb40cc4e3146ac9f3d3574a2c7a531" candidate_uuid: "07d06eb973b6438188660241a586e97b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a018f832db2a415989e154108749fd85"
I20260924 05:02:45.180801 19278 raft_consensus.cc:3037] T 4bcb40cc4e3146ac9f3d3574a2c7a531 P a018f832db2a415989e154108749fd85 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:02:45.180801 19147 raft_consensus.cc:3037] T 4bcb40cc4e3146ac9f3d3574a2c7a531 P d9b0d2d018c64b12b010d6f191631416 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:02:45.181516 19278 raft_consensus.cc:2454] T 4bcb40cc4e3146ac9f3d3574a2c7a531 P a018f832db2a415989e154108749fd85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07d06eb973b6438188660241a586e97b in term 1.
I20260924 05:02:45.181619 19147 raft_consensus.cc:2454] T 4bcb40cc4e3146ac9f3d3574a2c7a531 P d9b0d2d018c64b12b010d6f191631416 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07d06eb973b6438188660241a586e97b in term 1.
I20260924 05:02:45.181658 19343 leader_election.cc:302] T 4bcb40cc4e3146ac9f3d3574a2c7a531 P 07d06eb973b6438188660241a586e97b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07d06eb973b6438188660241a586e97b, a018f832db2a415989e154108749fd85; no voters:
I20260924 05:02:45.181739 19472 raft_consensus.cc:2781] T 4bcb40cc4e3146ac9f3d3574a2c7a531 P 07d06eb973b6438188660241a586e97b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:02:45.181890 19472 raft_consensus.cc:686] T 4bcb40cc4e3146ac9f3d3574a2c7a531 P 07d06eb973b6438188660241a586e97b [term 1 LEADER]: Becoming Leader. State: Replica: 07d06eb973b6438188660241a586e97b, State: Running, Role: LEADER
I20260924 05:02:45.181993 19472 consensus_queue.cc:237] T 4bcb40cc4e3146ac9f3d3574a2c7a531 P 07d06eb973b6438188660241a586e97b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9b0d2d018c64b12b010d6f191631416" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46273 } } peers { permanent_uuid: "07d06eb973b6438188660241a586e97b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41753 } } peers { permanent_uuid: "a018f832db2a415989e154108749fd85" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42175 } }
I20260924 05:02:45.182760 19005 catalog_manager.cc:5495] T 4bcb40cc4e3146ac9f3d3574a2c7a531 P 07d06eb973b6438188660241a586e97b reported cstate change: term changed from 0 to 1, leader changed from <none> to 07d06eb973b6438188660241a586e97b (127.0.98.3). New cstate: current_term: 1 leader_uuid: "07d06eb973b6438188660241a586e97b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9b0d2d018c64b12b010d6f191631416" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46273 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07d06eb973b6438188660241a586e97b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41753 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a018f832db2a415989e154108749fd85" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42175 } health_report { overall_health: UNKNOWN } } }
I20260924 05:02:45.197501 19466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:45.195197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53644 (local address 127.0.98.3:41753)
0924 05:02:45.195247 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 05:02:45.195249 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:02:45.195256 (+ 7us) server_negotiation.cc:407] Connection header received
0924 05:02:45.195292 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:02:45.195293 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:02:45.195314 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:02:45.195379 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:02:45.195497 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:45.195981 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:45.196723 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:45.196936 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:45.196989 (+ 53us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:45.197123 (+ 134us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:02:45.197222 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:02:45.197306 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 05:02:45.197367 (+ 61us) server_negotiation.cc:299] Negotiation successful
0924 05:02:45.197376 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 05:02:45.199200 19278 raft_consensus.cc:1261] T 4bcb40cc4e3146ac9f3d3574a2c7a531 P a018f832db2a415989e154108749fd85 [term 1 FOLLOWER]: Refusing update from remote peer 07d06eb973b6438188660241a586e97b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:02:45.199200 19147 raft_consensus.cc:1261] T 4bcb40cc4e3146ac9f3d3574a2c7a531 P d9b0d2d018c64b12b010d6f191631416 [term 1 FOLLOWER]: Refusing update from remote peer 07d06eb973b6438188660241a586e97b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:02:45.199558 19472 consensus_queue.cc:1035] T 4bcb40cc4e3146ac9f3d3574a2c7a531 P 07d06eb973b6438188660241a586e97b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9b0d2d018c64b12b010d6f191631416" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:02:45.199687 19479 consensus_queue.cc:1035] T 4bcb40cc4e3146ac9f3d3574a2c7a531 P 07d06eb973b6438188660241a586e97b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a018f832db2a415989e154108749fd85" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:02:45.202139 19484 mvcc.cc:205] Tried to move back new op lower bound from 7332766372653428736 to 7332766372586115072. Current Snapshot: MvccSnapshot[applied={T|T < 7332766372653428736}]
I20260924 05:02:45.202478 19482 mvcc.cc:205] Tried to move back new op lower bound from 7332766372653428736 to 7332766372586115072. Current Snapshot: MvccSnapshot[applied={T|T < 7332766372653428736}]
I20260924 05:02:45.204643 19005 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53154:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:02:45.204739 19005 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53154:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:02:45.205502 19005 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 204e19aaf6854ea896e91deb5d0a2cba: Sending DeleteTablet for 3 replicas of tablet 4bcb40cc4e3146ac9f3d3574a2c7a531
I20260924 05:02:45.205812 19127 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bcb40cc4e3146ac9f3d3574a2c7a531 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:02:45 UTC) from {username='slave'} at 127.0.0.1:41852
I20260924 05:02:45.205832 19389 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bcb40cc4e3146ac9f3d3574a2c7a531 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:02:45 UTC) from {username='slave'} at 127.0.0.1:53638
I20260924 05:02:45.206084 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 19065
I20260924 05:02:45.206163 19492 tablet_replica.cc:330] T 4bcb40cc4e3146ac9f3d3574a2c7a531 P 07d06eb973b6438188660241a586e97b: stopping tablet replica
I20260924 05:02:45.206274 19492 raft_consensus.cc:2229] T 4bcb40cc4e3146ac9f3d3574a2c7a531 P 07d06eb973b6438188660241a586e97b [term 1 LEADER]: Raft consensus shutting down.
I20260924 05:02:45.206303 19258 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bcb40cc4e3146ac9f3d3574a2c7a531 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:02:45 UTC) from {username='slave'} at 127.0.0.1:59092
I20260924 05:02:45.206458 19492 raft_consensus.cc:2258] T 4bcb40cc4e3146ac9f3d3574a2c7a531 P 07d06eb973b6438188660241a586e97b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:02:45.207012 19492 ts_tablet_manager.cc:1905] T 4bcb40cc4e3146ac9f3d3574a2c7a531 P 07d06eb973b6438188660241a586e97b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:02:45.208681 19492 ts_tablet_manager.cc:1918] T 4bcb40cc4e3146ac9f3d3574a2c7a531 P 07d06eb973b6438188660241a586e97b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 05:02:45.208750 19492 log.cc:1196] T 4bcb40cc4e3146ac9f3d3574a2c7a531 P 07d06eb973b6438188660241a586e97b: Deleting WAL directory at /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790225929848825-392-0/minicluster-data/ts-2/wal/wals/4bcb40cc4e3146ac9f3d3574a2c7a531
I20260924 05:02:45.208974 19492 ts_tablet_manager.cc:1939] T 4bcb40cc4e3146ac9f3d3574a2c7a531 P 07d06eb973b6438188660241a586e97b: Deleting consensus metadata
I20260924 05:02:45.209481 18992 catalog_manager.cc:4841] TS 07d06eb973b6438188660241a586e97b (127.0.98.3:41753): tablet 4bcb40cc4e3146ac9f3d3574a2c7a531 (table test-table [id=2c7fce36d4d243f1a5f6d113374ab80b]) successfully deleted
W20260924 05:02:45.212198 18991 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46273 (error 108)
I20260924 05:02:45.212244 19493 tablet_replica.cc:330] T 4bcb40cc4e3146ac9f3d3574a2c7a531 P a018f832db2a415989e154108749fd85: stopping tablet replica
I20260924 05:02:45.212338 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 19196
I20260924 05:02:45.212349 19493 raft_consensus.cc:2229] T 4bcb40cc4e3146ac9f3d3574a2c7a531 P a018f832db2a415989e154108749fd85 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:02:45.212457 19493 raft_consensus.cc:2258] T 4bcb40cc4e3146ac9f3d3574a2c7a531 P a018f832db2a415989e154108749fd85 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:02:45.212807 19465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:45.212706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46273 (local address 127.0.0.1:41862)
0924 05:02:45.212747 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:45.212776 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260924 05:02:45.212958 18991 catalog_manager.cc:4584] TS d9b0d2d018c64b12b010d6f191631416 (127.0.98.1:46273): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4bcb40cc4e3146ac9f3d3574a2c7a531: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46273: connect: Connection refused (error 111)
I20260924 05:02:45.217756 19465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:45.217648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42175 (local address 127.0.0.1:59104)
0924 05:02:45.217695 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:45.217721 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:42175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260924 05:02:45.217782 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 19327
I20260924 05:02:45.222311 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 18974
2026-09-24T05:02:45Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (693 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225929848825-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:02:45 dist-test-slave-70st krb5kdc[19497](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:02:45 dist-test-slave-70st krb5kdc[19497](info): set up 2 sockets
Sep 24 05:02:45 dist-test-slave-70st krb5kdc[19497](info): commencing operation
krb5kdc: starting...
W20260924 05:02:46.173007 392 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225929848825-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225929848825-392-0/krb5kdc/test-user.keytab.
Sep 24 05:02:46 dist-test-slave-70st krb5kdc[19497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226166, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:02:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:02:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 05:02:46.284679 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225929848825-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225929848825-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225929848825-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42775
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42775
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:02:46.364739 19510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:02:46.364955 19510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:02:46.364985 19510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:02:46.366145 19510 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:02:46.366186 19510 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:02:46.366204 19510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:02:46.366220 19510 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:02:46.366235 19510 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:02:46.367666 19510 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225929848825-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225929848825-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42775
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19510
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225929848825-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:02:46.367904 19510 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:02:46.368096 19510 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 05:02:46.372772 19510 server_base.cc:1034] running on GCE node
W20260924 05:02:46.372725 19515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:02:46.372710 19516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:02:46.372723 19518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:02:46.373224 19510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:02:46.373528 19510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:02:46.374655 19510 hybrid_clock.cc:648] HybridClock initialized: now 1790226166374646 us; error 28 us; skew 500 ppm
Sep 24 05:02:46 dist-test-slave-70st krb5kdc[19497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226166, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:02:46.377056 19510 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 05:02:46.377478 19510 webserver.cc:460] Webserver started at http://127.0.98.62:44119/ using document root <none> and password file <none>
I20260924 05:02:46.377660 19510 fs_manager.cc:359] Metadata directory not provided
I20260924 05:02:46.377697 19510 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:02:46.377799 19510 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:02:46.378803 19510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225929848825-392-0/minicluster-data/master-0/data/instance:
uuid: "b0d0a9b4d4064a9f9053b85fe1e59d07"
format_stamp: "Formatted at 2026-09-24 05:02:46 on dist-test-slave-70st"
server_key: "7e32a1b8cd8433dfb4bd0b07cac3d2d4"
server_key_iv: "b6d84596bb00464fc53ed0764e6af1d4"
server_key_version: "encryptionkey@0"
I20260924 05:02:46.379084 19510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225929848825-392-0/minicluster-data/master-0/wal/instance:
uuid: "b0d0a9b4d4064a9f9053b85fe1e59d07"
format_stamp: "Formatted at 2026-09-24 05:02:46 on dist-test-slave-70st"
server_key: "7e32a1b8cd8433dfb4bd0b07cac3d2d4"
server_key_iv: "b6d84596bb00464fc53ed0764e6af1d4"
server_key_version: "encryptionkey@0"
I20260924 05:02:46.380228 19510 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 05:02:46.380872 19525 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:02:46.381029 19510 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 05:02:46.381114 19510 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225929848825-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225929848825-392-0/minicluster-data/master-0/wal
uuid: "b0d0a9b4d4064a9f9053b85fe1e59d07"
format_stamp: "Formatted at 2026-09-24 05:02:46 on dist-test-slave-70st"
server_key: "7e32a1b8cd8433dfb4bd0b07cac3d2d4"
server_key_iv: "b6d84596bb00464fc53ed0764e6af1d4"
server_key_version: "encryptionkey@0"
I20260924 05:02:46.381166 19510 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225929848825-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225929848825-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225929848825-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:02:46.394898 19510 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:02:46.396934 19510 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:02:46.397048 19510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:02:46.400657 19510 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42775
I20260924 05:02:46.400698 19577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42775 every 8 connection(s)
I20260924 05:02:46.401016 19510 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:02:46.401881 19578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:02:46.404023 19578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0d0a9b4d4064a9f9053b85fe1e59d07: Bootstrap starting.
I20260924 05:02:46.404538 19578 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b0d0a9b4d4064a9f9053b85fe1e59d07: Neither blocks nor log segments found. Creating new log.
I20260924 05:02:46.404786 19578 log.cc:824] T 00000000000000000000000000000000 P b0d0a9b4d4064a9f9053b85fe1e59d07: Log is configured to *not* fsync() on all Append() calls
I20260924 05:02:46.405445 19578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0d0a9b4d4064a9f9053b85fe1e59d07: No bootstrap required, opened a new log
I20260924 05:02:46.406972 19578 raft_consensus.cc:348] T 00000000000000000000000000000000 P b0d0a9b4d4064a9f9053b85fe1e59d07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0d0a9b4d4064a9f9053b85fe1e59d07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42775 } }
I20260924 05:02:46.407102 19578 raft_consensus.cc:374] T 00000000000000000000000000000000 P b0d0a9b4d4064a9f9053b85fe1e59d07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:02:46.407122 19578 raft_consensus.cc:729] T 00000000000000000000000000000000 P b0d0a9b4d4064a9f9053b85fe1e59d07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0d0a9b4d4064a9f9053b85fe1e59d07, State: Initialized, Role: FOLLOWER
I20260924 05:02:46.407256 19578 consensus_queue.cc:260] T 00000000000000000000000000000000 P b0d0a9b4d4064a9f9053b85fe1e59d07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0d0a9b4d4064a9f9053b85fe1e59d07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42775 } }
I20260924 05:02:46.407320 19578 raft_consensus.cc:388] T 00000000000000000000000000000000 P b0d0a9b4d4064a9f9053b85fe1e59d07 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:02:46.407342 19578 raft_consensus.cc:482] T 00000000000000000000000000000000 P b0d0a9b4d4064a9f9053b85fe1e59d07 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:02:46.407384 19578 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b0d0a9b4d4064a9f9053b85fe1e59d07 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:02:46.407855 19578 raft_consensus.cc:504] T 00000000000000000000000000000000 P b0d0a9b4d4064a9f9053b85fe1e59d07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0d0a9b4d4064a9f9053b85fe1e59d07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42775 } }
I20260924 05:02:46.407938 19578 leader_election.cc:302] T 00000000000000000000000000000000 P b0d0a9b4d4064a9f9053b85fe1e59d07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b0d0a9b4d4064a9f9053b85fe1e59d07; no voters:
I20260924 05:02:46.408083 19578 leader_election.cc:288] T 00000000000000000000000000000000 P b0d0a9b4d4064a9f9053b85fe1e59d07 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:02:46.408120 19581 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b0d0a9b4d4064a9f9053b85fe1e59d07 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:02:46.408282 19581 raft_consensus.cc:686] T 00000000000000000000000000000000 P b0d0a9b4d4064a9f9053b85fe1e59d07 [term 1 LEADER]: Becoming Leader. State: Replica: b0d0a9b4d4064a9f9053b85fe1e59d07, State: Running, Role: LEADER
I20260924 05:02:46.408313 19578 sys_catalog.cc:564] T 00000000000000000000000000000000 P b0d0a9b4d4064a9f9053b85fe1e59d07 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:02:46.408389 19581 consensus_queue.cc:237] T 00000000000000000000000000000000 P b0d0a9b4d4064a9f9053b85fe1e59d07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0d0a9b4d4064a9f9053b85fe1e59d07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42775 } }
I20260924 05:02:46.408907 19582 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0d0a9b4d4064a9f9053b85fe1e59d07 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b0d0a9b4d4064a9f9053b85fe1e59d07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0d0a9b4d4064a9f9053b85fe1e59d07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42775 } } }
I20260924 05:02:46.408988 19582 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0d0a9b4d4064a9f9053b85fe1e59d07 [sys.catalog]: This master's current role is: LEADER
I20260924 05:02:46.409283 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 19510
I20260924 05:02:46.409376 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225929848825-392-0/minicluster-data/master-0/wal/instance
I20260924 05:02:46.409555 392 external_mini_cluster.cc:1428] Setting key 54188b92e7ae19f59e97212de0e9f8fe
I20260924 05:02:46.409758 19583 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0d0a9b4d4064a9f9053b85fe1e59d07 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b0d0a9b4d4064a9f9053b85fe1e59d07. Latest consensus state: current_term: 1 leader_uuid: "b0d0a9b4d4064a9f9053b85fe1e59d07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0d0a9b4d4064a9f9053b85fe1e59d07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42775 } } }
W20260924 05:02:46.409996 19597 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b0d0a9b4d4064a9f9053b85fe1e59d07: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260924 05:02:46.410211 19583 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0d0a9b4d4064a9f9053b85fe1e59d07 [sys.catalog]: This master's current role is: LEADER
W20260924 05:02:46.410220 19597 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 05:02:46.410308 19590 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:02:46.410483 19590 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:02:46.411850 19590 catalog_manager.cc:1347] Generated new cluster ID: 63bf287c074e45cab4a5883b0681590d
I20260924 05:02:46.411895 19590 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 05:02:46 dist-test-slave-70st krb5kdc[19497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226166, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:02:46.420730 19600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:46.410007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43980 (local address 127.0.98.62:42775)
0924 05:02:46.410196 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 05:02:46.410205 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 05:02:46.410240 (+ 35us) server_negotiation.cc:407] Connection header received
0924 05:02:46.410475 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:02:46.410483 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:02:46.410542 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:02:46.410680 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:02:46.411365 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:46.412616 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:46.413701 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:46.414315 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:46.414420 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:46.417124 (+ 2704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:02:46.417128 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:02:46.417128 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:02:46.417140 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:02:46.419116 (+ 1976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:02:46.419525 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:02:46.419528 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:02:46.419528 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:02:46.419641 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:02:46.419853 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:02:46.419854 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:02:46.419854 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:02:46.420142 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:02:46.420271 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0924 05:02:46.420386 (+ 115us) server_negotiation.cc:299] Negotiation successful
0924 05:02:46.420514 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":110,"threads_started":1}
I20260924 05:02:46.428915 19590 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:02:46.429584 19590 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:02:46.431792 19590 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b0d0a9b4d4064a9f9053b85fe1e59d07: Generated new TSK 0
I20260924 05:02:46.431967 19590 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225929848825-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225929848825-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 05:02:46.460703 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225929848825-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225929848825-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225929848825-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225929848825-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42775
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225929848825-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:02:46.539456 19604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:02:46.539656 19604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:02:46.539678 19604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:02:46.540802 19604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:02:46.540843 19604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:02:46.540867 19604 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 05:02:46.542393 19604 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225929848825-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225929848825-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225929848825-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225929848825-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19604
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225929848825-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:02:46.542614 19604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:02:46.542830 19604 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:02:46.547475 19609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:02:46.547573 19610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:02:46.547580 19604 server_base.cc:1034] running on GCE node
W20260924 05:02:46.547747 19612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:02:46.547935 19604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:02:46.548178 19604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:02:46.549314 19604 hybrid_clock.cc:648] HybridClock initialized: now 1790226166549305 us; error 30 us; skew 500 ppm
Sep 24 05:02:46 dist-test-slave-70st krb5kdc[19497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226166, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:02:46.551692 19604 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 05:02:46.552101 19604 webserver.cc:460] Webserver started at http://127.0.98.1:44265/ using document root <none> and password file <none>
I20260924 05:02:46.552278 19604 fs_manager.cc:359] Metadata directory not provided
I20260924 05:02:46.552317 19604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:02:46.552412 19604 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:02:46.553380 19604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225929848825-392-0/minicluster-data/ts-0/data/instance:
uuid: "b611e8f4f86d4d219aab4c221b1b0b37"
format_stamp: "Formatted at 2026-09-24 05:02:46 on dist-test-slave-70st"
server_key: "f389a7d6b6b3b8d19f22df9a0de7bf14"
server_key_iv: "ca18f731eefda77c133d019a27c49fb0"
server_key_version: "encryptionkey@0"
I20260924 05:02:46.553714 19604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225929848825-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b611e8f4f86d4d219aab4c221b1b0b37"
format_stamp: "Formatted at 2026-09-24 05:02:46 on dist-test-slave-70st"
server_key: "f389a7d6b6b3b8d19f22df9a0de7bf14"
server_key_iv: "ca18f731eefda77c133d019a27c49fb0"
server_key_version: "encryptionkey@0"
I20260924 05:02:46.554931 19604 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 05:02:46.555655 19619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:02:46.555864 19604 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 05:02:46.555958 19604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225929848825-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225929848825-392-0/minicluster-data/ts-0/wal
uuid: "b611e8f4f86d4d219aab4c221b1b0b37"
format_stamp: "Formatted at 2026-09-24 05:02:46 on dist-test-slave-70st"
server_key: "f389a7d6b6b3b8d19f22df9a0de7bf14"
server_key_iv: "ca18f731eefda77c133d019a27c49fb0"
server_key_version: "encryptionkey@0"
I20260924 05:02:46.556035 19604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225929848825-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225929848825-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225929848825-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:02:46.573748 19604 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:02:46.576057 19604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:02:46.576186 19604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:02:46.576432 19604 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:02:46.576756 19604 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:02:46.576794 19604 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:02:46.576820 19604 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:02:46.576848 19604 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:02:46.582856 19604 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37945
I20260924 05:02:46.582921 19732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37945 every 8 connection(s)
I20260924 05:02:46.583223 19604 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225929848825-392-0/minicluster-data/ts-0/data/info.pb
I20260924 05:02:46.585117 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 19604
I20260924 05:02:46.585323 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225929848825-392-0/minicluster-data/ts-0/wal/instance
I20260924 05:02:46.585546 392 external_mini_cluster.cc:1428] Setting key d9a38dfc9c9992fbb508f5b027cd953e
Sep 24 05:02:46 dist-test-slave-70st krb5kdc[19497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226166, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:02:46.593910 19600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:46.583937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53799 (local address 127.0.98.62:42775)
0924 05:02:46.583977 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 05:02:46.583979 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:02:46.584165 (+ 186us) server_negotiation.cc:407] Connection header received
0924 05:02:46.585166 (+ 1001us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:02:46.585167 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:02:46.585184 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:02:46.585236 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:02:46.585975 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:46.586545 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:46.587528 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:46.587734 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:46.587789 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:46.589811 (+ 2022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:02:46.589812 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:02:46.589813 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:02:46.589822 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:02:46.591988 (+ 2166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:02:46.592530 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:02:46.592532 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:02:46.592532 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:02:46.592634 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:02:46.593107 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:02:46.593108 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:02:46.593109 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:02:46.593268 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:02:46.593319 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 05:02:46.593764 (+ 445us) server_negotiation.cc:299] Negotiation successful
0924 05:02:46.593842 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 05:02:46.594173 19735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:46.583924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42775 (local address 127.0.98.1:53799)
0924 05:02:46.584055 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:46.584090 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 05:02:46.585067 (+ 977us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:02:46.585309 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:02:46.585312 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:02:46.585614 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:02:46.585895 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:46.585901 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:46.586625 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:46.586626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:46.587447 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:46.587450 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:46.587806 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:46.587807 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:46.587941 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:46.588410 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:02:46.588418 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:02:46.589716 (+ 1298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:02:46.592098 (+ 2382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:02:46.592106 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:02:46.592108 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:02:46.592417 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:02:46.592742 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:02:46.592743 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:02:46.592743 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:02:46.592975 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:02:46.593380 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:02:46.593388 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:02:46.593600 (+ 212us) client_negotiation.cc:769] Sending connection context
0924 05:02:46.593743 (+ 143us) client_negotiation.cc:240] Negotiation successful
0924 05:02:46.593928 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":69,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 05:02:46.594769 19733 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42775
I20260924 05:02:46.594870 19733 heartbeater.cc:464] Registering TS with master...
I20260924 05:02:46.595057 19733 heartbeater.cc:511] Master 127.0.98.62:42775 requested a full tablet report, sending...
I20260924 05:02:46.595690 19542 ts_manager.cc:194] Registered new tserver with Master: b611e8f4f86d4d219aab4c221b1b0b37 (127.0.98.1:37945)
I20260924 05:02:46.596354 19542 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:53799
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225929848825-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225929848825-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 05:02:46.610213 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225929848825-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225929848825-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225929848825-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225929848825-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42775
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225929848825-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:02:46.688211 19738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:02:46.688412 19738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:02:46.688443 19738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:02:46.689584 19738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:02:46.689621 19738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:02:46.689656 19738 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 05:02:46.691085 19738 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225929848825-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225929848825-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225929848825-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225929848825-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19738
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225929848825-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:02:46.691315 19738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:02:46.691509 19738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:02:46.696188 19746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:02:46.696206 19743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:02:46.696226 19744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:02:46.696521 19738 server_base.cc:1034] running on GCE node
I20260924 05:02:46.696725 19738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:02:46.696940 19738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:02:46.698081 19738 hybrid_clock.cc:648] HybridClock initialized: now 1790226166698067 us; error 28 us; skew 500 ppm
Sep 24 05:02:46 dist-test-slave-70st krb5kdc[19497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226166, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:02:46.700416 19738 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 05:02:46.700819 19738 webserver.cc:460] Webserver started at http://127.0.98.2:46573/ using document root <none> and password file <none>
I20260924 05:02:46.701005 19738 fs_manager.cc:359] Metadata directory not provided
I20260924 05:02:46.701047 19738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:02:46.701148 19738 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:02:46.702199 19738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225929848825-392-0/minicluster-data/ts-1/data/instance:
uuid: "07927f799ebf4bac807617b5404c7547"
format_stamp: "Formatted at 2026-09-24 05:02:46 on dist-test-slave-70st"
server_key: "f9fef0196c3461b698ae77bae688f8eb"
server_key_iv: "487f5d3eb0311e0aaf2d7d7f797d3094"
server_key_version: "encryptionkey@0"
I20260924 05:02:46.702525 19738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225929848825-392-0/minicluster-data/ts-1/wal/instance:
uuid: "07927f799ebf4bac807617b5404c7547"
format_stamp: "Formatted at 2026-09-24 05:02:46 on dist-test-slave-70st"
server_key: "f9fef0196c3461b698ae77bae688f8eb"
server_key_iv: "487f5d3eb0311e0aaf2d7d7f797d3094"
server_key_version: "encryptionkey@0"
I20260924 05:02:46.703737 19738 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 05:02:46.704509 19753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:02:46.704757 19738 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 05:02:46.704838 19738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225929848825-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225929848825-392-0/minicluster-data/ts-1/wal
uuid: "07927f799ebf4bac807617b5404c7547"
format_stamp: "Formatted at 2026-09-24 05:02:46 on dist-test-slave-70st"
server_key: "f9fef0196c3461b698ae77bae688f8eb"
server_key_iv: "487f5d3eb0311e0aaf2d7d7f797d3094"
server_key_version: "encryptionkey@0"
I20260924 05:02:46.704907 19738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225929848825-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225929848825-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225929848825-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:02:46.721072 19738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:02:46.723033 19738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:02:46.723151 19738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:02:46.723369 19738 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:02:46.723671 19738 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:02:46.723704 19738 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:02:46.723732 19738 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:02:46.723755 19738 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:02:46.729082 19738 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35961
I20260924 05:02:46.729138 19866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35961 every 8 connection(s)
I20260924 05:02:46.729458 19738 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225929848825-392-0/minicluster-data/ts-1/data/info.pb
I20260924 05:02:46.734686 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 19738
I20260924 05:02:46.734817 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225929848825-392-0/minicluster-data/ts-1/wal/instance
I20260924 05:02:46.735021 392 external_mini_cluster.cc:1428] Setting key d3d4da33461e4b9cb2845d90cca2d2c1
Sep 24 05:02:46 dist-test-slave-70st krb5kdc[19497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226166, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:02:46.738512 19600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:46.730141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52471 (local address 127.0.98.62:42775)
0924 05:02:46.730236 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0924 05:02:46.730239 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:02:46.730426 (+ 187us) server_negotiation.cc:407] Connection header received
0924 05:02:46.731370 (+ 944us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:02:46.731372 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:02:46.731385 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:02:46.731432 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:02:46.732072 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:46.732575 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:46.733581 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:46.733769 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:46.733828 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:46.736097 (+ 2269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:02:46.736098 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:02:46.736098 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:02:46.736105 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:02:46.737242 (+ 1137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:02:46.737642 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:02:46.737643 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:02:46.737644 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:02:46.737697 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:02:46.737905 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:02:46.737905 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:02:46.737906 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:02:46.738015 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:02:46.738069 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 05:02:46.738318 (+ 249us) server_negotiation.cc:299] Negotiation successful
0924 05:02:46.738424 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 05:02:46.738655 19869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:46.730100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42775 (local address 127.0.98.2:52471)
0924 05:02:46.730300 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:46.730341 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 05:02:46.731255 (+ 914us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:02:46.731515 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:02:46.731519 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:02:46.731791 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:02:46.731980 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:46.731985 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:46.732655 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:46.732656 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:46.733500 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:46.733502 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:46.733838 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:46.733839 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:46.733956 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:46.734429 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:02:46.734437 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:02:46.736007 (+ 1570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:02:46.737315 (+ 1308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:02:46.737321 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:02:46.737323 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:02:46.737580 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:02:46.737750 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:02:46.737751 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:02:46.737751 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:02:46.737852 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:02:46.738066 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:02:46.738072 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:02:46.738207 (+ 135us) client_negotiation.cc:769] Sending connection context
0924 05:02:46.738293 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 05:02:46.738446 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":75,"threads_started":1}
I20260924 05:02:46.739055 19867 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42775
I20260924 05:02:46.739161 19867 heartbeater.cc:464] Registering TS with master...
I20260924 05:02:46.739357 19867 heartbeater.cc:511] Master 127.0.98.62:42775 requested a full tablet report, sending...
I20260924 05:02:46.739850 19542 ts_manager.cc:194] Registered new tserver with Master: 07927f799ebf4bac807617b5404c7547 (127.0.98.2:35961)
I20260924 05:02:46.740342 19542 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:52471
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225929848825-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225929848825-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 05:02:46.760048 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225929848825-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225929848825-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225929848825-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225929848825-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42775
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225929848825-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:02:46.837599 19872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:02:46.837790 19872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:02:46.837811 19872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:02:46.838876 19872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:02:46.838910 19872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:02:46.838934 19872 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 05:02:46.840313 19872 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225929848825-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225929848825-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225929848825-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225929848825-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19872
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225929848825-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:02:46.840489 19872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:02:46.840662 19872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:02:46.845228 19880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:02:46.845388 19872 server_base.cc:1034] running on GCE node
W20260924 05:02:46.845242 19877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:02:46.845268 19878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:02:46.845686 19872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:02:46.845892 19872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:02:46.847043 19872 hybrid_clock.cc:648] HybridClock initialized: now 1790226166847027 us; error 34 us; skew 500 ppm
Sep 24 05:02:46 dist-test-slave-70st krb5kdc[19497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226166, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:02:46.849383 19872 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 05:02:46.849772 19872 webserver.cc:460] Webserver started at http://127.0.98.3:38019/ using document root <none> and password file <none>
I20260924 05:02:46.849956 19872 fs_manager.cc:359] Metadata directory not provided
I20260924 05:02:46.849994 19872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:02:46.850080 19872 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:02:46.850993 19872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225929848825-392-0/minicluster-data/ts-2/data/instance:
uuid: "1a50d2ffa7fe40a5a4e3b5759ac910a8"
format_stamp: "Formatted at 2026-09-24 05:02:46 on dist-test-slave-70st"
server_key: "0d6e11e125d27e3d037fde29f56bdec3"
server_key_iv: "05ced243ebb06956b04383e2c0631794"
server_key_version: "encryptionkey@0"
I20260924 05:02:46.851254 19872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225929848825-392-0/minicluster-data/ts-2/wal/instance:
uuid: "1a50d2ffa7fe40a5a4e3b5759ac910a8"
format_stamp: "Formatted at 2026-09-24 05:02:46 on dist-test-slave-70st"
server_key: "0d6e11e125d27e3d037fde29f56bdec3"
server_key_iv: "05ced243ebb06956b04383e2c0631794"
server_key_version: "encryptionkey@0"
I20260924 05:02:46.852428 19872 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 05:02:46.853016 19887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:02:46.853153 19872 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:02:46.853222 19872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225929848825-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225929848825-392-0/minicluster-data/ts-2/wal
uuid: "1a50d2ffa7fe40a5a4e3b5759ac910a8"
format_stamp: "Formatted at 2026-09-24 05:02:46 on dist-test-slave-70st"
server_key: "0d6e11e125d27e3d037fde29f56bdec3"
server_key_iv: "05ced243ebb06956b04383e2c0631794"
server_key_version: "encryptionkey@0"
I20260924 05:02:46.853284 19872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225929848825-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225929848825-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225929848825-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:02:46.868276 19872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:02:46.870198 19872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:02:46.870318 19872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:02:46.870541 19872 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:02:46.870814 19872 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:02:46.870846 19872 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:02:46.870867 19872 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:02:46.870889 19872 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:02:46.876312 19872 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40493
I20260924 05:02:46.876390 20000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40493 every 8 connection(s)
I20260924 05:02:46.876658 19872 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225929848825-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 05:02:46 dist-test-slave-70st krb5kdc[19497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226166, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:02:46.884578 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 19872
I20260924 05:02:46.884683 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225929848825-392-0/minicluster-data/ts-2/wal/instance
I20260924 05:02:46.884843 392 external_mini_cluster.cc:1428] Setting key 27443bcb0ff854172955f403df41f4e9
I20260924 05:02:46.884900 19600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:46.877296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60695 (local address 127.0.98.62:42775)
0924 05:02:46.877357 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 05:02:46.877359 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:02:46.877543 (+ 184us) server_negotiation.cc:407] Connection header received
0924 05:02:46.878404 (+ 861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:02:46.878405 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:02:46.878414 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:02:46.878466 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:02:46.879003 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:46.879480 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:46.880400 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:46.880580 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:46.880627 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:46.882500 (+ 1873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:02:46.882502 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:02:46.882502 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:02:46.882508 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:02:46.883700 (+ 1192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:02:46.884035 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:02:46.884036 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:02:46.884037 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:02:46.884086 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:02:46.884265 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:02:46.884265 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:02:46.884265 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:02:46.884375 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:02:46.884426 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 05:02:46.884698 (+ 272us) server_negotiation.cc:299] Negotiation successful
0924 05:02:46.884777 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 05:02:46.885015 20003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:46.877261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42775 (local address 127.0.98.3:60695)
0924 05:02:46.877404 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:46.877468 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0924 05:02:46.878319 (+ 851us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:02:46.878518 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:02:46.878521 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:02:46.878755 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:02:46.878939 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:46.878943 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:46.879541 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:46.879541 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:46.880316 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:46.880319 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:46.880634 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:46.880635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:46.880732 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:46.881113 (+ 381us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:02:46.881120 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:02:46.882415 (+ 1295us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:02:46.883757 (+ 1342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:02:46.883763 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:02:46.883765 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:02:46.883973 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:02:46.884133 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:02:46.884134 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:02:46.884134 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:02:46.884223 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:02:46.884423 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:02:46.884456 (+ 33us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:02:46.884573 (+ 117us) client_negotiation.cc:769] Sending connection context
0924 05:02:46.884660 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 05:02:46.884800 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":70,"threads_started":1}
I20260924 05:02:46.885459 20001 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42775
I20260924 05:02:46.885609 20001 heartbeater.cc:464] Registering TS with master...
I20260924 05:02:46.885783 20001 heartbeater.cc:511] Master 127.0.98.62:42775 requested a full tablet report, sending...
I20260924 05:02:46.886107 19542 ts_manager.cc:194] Registered new tserver with Master: 1a50d2ffa7fe40a5a4e3b5759ac910a8 (127.0.98.3:40493)
I20260924 05:02:46.886466 19542 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:60695
I20260924 05:02:46.896377 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 05:02:46.903046 19600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:46.897690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43994 (local address 127.0.98.62:42775)
0924 05:02:46.897736 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 05:02:46.897738 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:02:46.897822 (+ 84us) server_negotiation.cc:407] Connection header received
0924 05:02:46.897877 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:02:46.897878 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:02:46.897889 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:02:46.897946 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:02:46.898444 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:46.898907 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:46.899627 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:46.899795 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:46.899836 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:46.900955 (+ 1119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:02:46.900956 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:02:46.900956 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:02:46.900962 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:02:46.902203 (+ 1241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:02:46.902486 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:02:46.902487 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:02:46.902487 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:02:46.902530 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:02:46.902651 (+ 121us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:02:46.902652 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:02:46.902652 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:02:46.902773 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:02:46.902836 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 05:02:46.902923 (+ 87us) server_negotiation.cc:299] Negotiation successful
0924 05:02:46.902975 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 05:02:46.904459 19542 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43994:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 05:02:46.906050 19542 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 05:02:46.912019 20009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:46.908025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40493 (local address 127.0.0.1:42890)
0924 05:02:46.908298 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:46.908367 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0924 05:02:46.908432 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:02:46.908667 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:02:46.908668 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:02:46.908670 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:02:46.908885 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:46.908887 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:46.910129 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:46.910130 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:46.911061 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:46.911064 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:46.911731 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:46.911732 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:46.911814 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:46.911829 (+ 15us) client_negotiation.cc:769] Sending connection context
0924 05:02:46.911890 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 05:02:46.911938 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"mutex_wait_us":158,"thread_start_us":50,"threads_started":1}
I20260924 05:02:46.912149 20010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:46.908025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35961 (local address 127.0.0.1:54522)
0924 05:02:46.908320 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:46.908386 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0924 05:02:46.908440 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:02:46.908645 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:02:46.908648 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:02:46.908650 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:02:46.908747 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:46.908749 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:46.909932 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:46.909933 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:46.911085 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:46.911088 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:46.911901 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:46.911902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:46.911972 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:46.911976 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 05:02:46.912035 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 05:02:46.912073 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"mutex_wait_us":39,"thread_start_us":46,"threads_started":1}
I20260924 05:02:46.912307 20013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:46.908173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42890 (local address 127.0.98.3:40493)
0924 05:02:46.908402 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0924 05:02:46.908407 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 05:02:46.908417 (+ 10us) server_negotiation.cc:407] Connection header received
0924 05:02:46.908537 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:02:46.908539 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:02:46.908595 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:02:46.908663 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:02:46.908960 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:46.910026 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:46.911131 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:46.911654 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:46.911725 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:46.911781 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 05:02:46.912192 (+ 411us) server_negotiation.cc:299] Negotiation successful
0924 05:02:46.912234 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":71,"threads_started":1}
I20260924 05:02:46.912422 20011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:46.908104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54522 (local address 127.0.98.2:35961)
0924 05:02:46.908493 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0924 05:02:46.908497 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 05:02:46.908505 (+ 8us) server_negotiation.cc:407] Connection header received
0924 05:02:46.908537 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:02:46.908539 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:02:46.908555 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:02:46.908638 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:02:46.908811 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:46.909843 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:46.911178 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:46.911823 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:46.911891 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:46.911946 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 05:02:46.912051 (+ 105us) server_negotiation.cc:299] Negotiation successful
0924 05:02:46.912349 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":91,"threads_started":1}
I20260924 05:02:46.912523 20012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:46.908203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37945 (local address 127.0.0.1:41372)
0924 05:02:46.908757 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:46.908766 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:02:46.908812 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:02:46.909056 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:02:46.909057 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:02:46.909059 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:02:46.909098 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:46.909099 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:46.910389 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:46.910390 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:46.911448 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:46.911450 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:46.912322 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:46.912323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:46.912383 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:46.912387 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 05:02:46.912427 (+ 40us) client_negotiation.cc:240] Negotiation successful
0924 05:02:46.912460 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":45,"threads_started":1}
I20260924 05:02:46.912774 20014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:46.908364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41372 (local address 127.0.98.1:37945)
0924 05:02:46.908547 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 05:02:46.908552 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 05:02:46.908864 (+ 312us) server_negotiation.cc:407] Connection header received
0924 05:02:46.908908 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:02:46.908910 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:02:46.908923 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:02:46.908984 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:02:46.909175 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:46.910296 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:46.911521 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:46.912178 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:46.912239 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:46.912289 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 05:02:46.912660 (+ 371us) server_negotiation.cc:299] Negotiation successful
0924 05:02:46.912701 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":87,"threads_started":1}
I20260924 05:02:46.913025 19801 tablet_service.cc:1467] Processing CreateTablet for tablet f2b486dd27a34b3ea5ba4e265afe08de (DEFAULT_TABLE table=test-table [id=fcc5bc78cb7541f180969b228a37c541]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:02:46.912995 19935 tablet_service.cc:1467] Processing CreateTablet for tablet f2b486dd27a34b3ea5ba4e265afe08de (DEFAULT_TABLE table=test-table [id=fcc5bc78cb7541f180969b228a37c541]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:02:46.913326 19935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2b486dd27a34b3ea5ba4e265afe08de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:02:46.913326 19801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2b486dd27a34b3ea5ba4e265afe08de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:02:46.913431 19667 tablet_service.cc:1467] Processing CreateTablet for tablet f2b486dd27a34b3ea5ba4e265afe08de (DEFAULT_TABLE table=test-table [id=fcc5bc78cb7541f180969b228a37c541]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:02:46.913718 19667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2b486dd27a34b3ea5ba4e265afe08de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:02:46.916042 20015 tablet_bootstrap.cc:492] T f2b486dd27a34b3ea5ba4e265afe08de P b611e8f4f86d4d219aab4c221b1b0b37: Bootstrap starting.
I20260924 05:02:46.916388 20017 tablet_bootstrap.cc:492] T f2b486dd27a34b3ea5ba4e265afe08de P 1a50d2ffa7fe40a5a4e3b5759ac910a8: Bootstrap starting.
I20260924 05:02:46.916622 20016 tablet_bootstrap.cc:492] T f2b486dd27a34b3ea5ba4e265afe08de P 07927f799ebf4bac807617b5404c7547: Bootstrap starting.
I20260924 05:02:46.916785 20015 tablet_bootstrap.cc:654] T f2b486dd27a34b3ea5ba4e265afe08de P b611e8f4f86d4d219aab4c221b1b0b37: Neither blocks nor log segments found. Creating new log.
I20260924 05:02:46.916970 20017 tablet_bootstrap.cc:654] T f2b486dd27a34b3ea5ba4e265afe08de P 1a50d2ffa7fe40a5a4e3b5759ac910a8: Neither blocks nor log segments found. Creating new log.
I20260924 05:02:46.917243 20016 tablet_bootstrap.cc:654] T f2b486dd27a34b3ea5ba4e265afe08de P 07927f799ebf4bac807617b5404c7547: Neither blocks nor log segments found. Creating new log.
I20260924 05:02:46.917346 20015 log.cc:824] T f2b486dd27a34b3ea5ba4e265afe08de P b611e8f4f86d4d219aab4c221b1b0b37: Log is configured to *not* fsync() on all Append() calls
I20260924 05:02:46.917644 20017 log.cc:824] T f2b486dd27a34b3ea5ba4e265afe08de P 1a50d2ffa7fe40a5a4e3b5759ac910a8: Log is configured to *not* fsync() on all Append() calls
I20260924 05:02:46.917794 20016 log.cc:824] T f2b486dd27a34b3ea5ba4e265afe08de P 07927f799ebf4bac807617b5404c7547: Log is configured to *not* fsync() on all Append() calls
I20260924 05:02:46.918267 20015 tablet_bootstrap.cc:492] T f2b486dd27a34b3ea5ba4e265afe08de P b611e8f4f86d4d219aab4c221b1b0b37: No bootstrap required, opened a new log
I20260924 05:02:46.918341 20015 ts_tablet_manager.cc:1397] T f2b486dd27a34b3ea5ba4e265afe08de P b611e8f4f86d4d219aab4c221b1b0b37: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 05:02:46.918341 20017 tablet_bootstrap.cc:492] T f2b486dd27a34b3ea5ba4e265afe08de P 1a50d2ffa7fe40a5a4e3b5759ac910a8: No bootstrap required, opened a new log
I20260924 05:02:46.918416 20017 ts_tablet_manager.cc:1397] T f2b486dd27a34b3ea5ba4e265afe08de P 1a50d2ffa7fe40a5a4e3b5759ac910a8: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260924 05:02:46.918473 20016 tablet_bootstrap.cc:492] T f2b486dd27a34b3ea5ba4e265afe08de P 07927f799ebf4bac807617b5404c7547: No bootstrap required, opened a new log
I20260924 05:02:46.918522 20016 ts_tablet_manager.cc:1397] T f2b486dd27a34b3ea5ba4e265afe08de P 07927f799ebf4bac807617b5404c7547: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 05:02:46.919854 20017 raft_consensus.cc:348] T f2b486dd27a34b3ea5ba4e265afe08de P 1a50d2ffa7fe40a5a4e3b5759ac910a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a50d2ffa7fe40a5a4e3b5759ac910a8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40493 } } peers { permanent_uuid: "b611e8f4f86d4d219aab4c221b1b0b37" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37945 } } peers { permanent_uuid: "07927f799ebf4bac807617b5404c7547" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35961 } }
I20260924 05:02:46.919991 20017 raft_consensus.cc:374] T f2b486dd27a34b3ea5ba4e265afe08de P 1a50d2ffa7fe40a5a4e3b5759ac910a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:02:46.920022 20017 raft_consensus.cc:729] T f2b486dd27a34b3ea5ba4e265afe08de P 1a50d2ffa7fe40a5a4e3b5759ac910a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a50d2ffa7fe40a5a4e3b5759ac910a8, State: Initialized, Role: FOLLOWER
I20260924 05:02:46.920162 20017 consensus_queue.cc:260] T f2b486dd27a34b3ea5ba4e265afe08de P 1a50d2ffa7fe40a5a4e3b5759ac910a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a50d2ffa7fe40a5a4e3b5759ac910a8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40493 } } peers { permanent_uuid: "b611e8f4f86d4d219aab4c221b1b0b37" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37945 } } peers { permanent_uuid: "07927f799ebf4bac807617b5404c7547" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35961 } }
I20260924 05:02:46.920260 20015 raft_consensus.cc:348] T f2b486dd27a34b3ea5ba4e265afe08de P b611e8f4f86d4d219aab4c221b1b0b37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a50d2ffa7fe40a5a4e3b5759ac910a8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40493 } } peers { permanent_uuid: "b611e8f4f86d4d219aab4c221b1b0b37" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37945 } } peers { permanent_uuid: "07927f799ebf4bac807617b5404c7547" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35961 } }
I20260924 05:02:46.920403 20017 ts_tablet_manager.cc:1428] T f2b486dd27a34b3ea5ba4e265afe08de P 1a50d2ffa7fe40a5a4e3b5759ac910a8: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 05:02:46.920428 20015 raft_consensus.cc:374] T f2b486dd27a34b3ea5ba4e265afe08de P b611e8f4f86d4d219aab4c221b1b0b37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:02:46.920471 20015 raft_consensus.cc:729] T f2b486dd27a34b3ea5ba4e265afe08de P b611e8f4f86d4d219aab4c221b1b0b37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b611e8f4f86d4d219aab4c221b1b0b37, State: Initialized, Role: FOLLOWER
I20260924 05:02:46.920503 20001 heartbeater.cc:503] Master 127.0.98.62:42775 was elected leader, sending a full tablet report...
I20260924 05:02:46.920496 20016 raft_consensus.cc:348] T f2b486dd27a34b3ea5ba4e265afe08de P 07927f799ebf4bac807617b5404c7547 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a50d2ffa7fe40a5a4e3b5759ac910a8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40493 } } peers { permanent_uuid: "b611e8f4f86d4d219aab4c221b1b0b37" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37945 } } peers { permanent_uuid: "07927f799ebf4bac807617b5404c7547" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35961 } }
I20260924 05:02:46.920578 20015 consensus_queue.cc:260] T f2b486dd27a34b3ea5ba4e265afe08de P b611e8f4f86d4d219aab4c221b1b0b37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a50d2ffa7fe40a5a4e3b5759ac910a8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40493 } } peers { permanent_uuid: "b611e8f4f86d4d219aab4c221b1b0b37" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37945 } } peers { permanent_uuid: "07927f799ebf4bac807617b5404c7547" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35961 } }
I20260924 05:02:46.920655 20016 raft_consensus.cc:374] T f2b486dd27a34b3ea5ba4e265afe08de P 07927f799ebf4bac807617b5404c7547 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:02:46.920694 20016 raft_consensus.cc:729] T f2b486dd27a34b3ea5ba4e265afe08de P 07927f799ebf4bac807617b5404c7547 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07927f799ebf4bac807617b5404c7547, State: Initialized, Role: FOLLOWER
I20260924 05:02:46.920797 20016 consensus_queue.cc:260] T f2b486dd27a34b3ea5ba4e265afe08de P 07927f799ebf4bac807617b5404c7547 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a50d2ffa7fe40a5a4e3b5759ac910a8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40493 } } peers { permanent_uuid: "b611e8f4f86d4d219aab4c221b1b0b37" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37945 } } peers { permanent_uuid: "07927f799ebf4bac807617b5404c7547" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35961 } }
I20260924 05:02:46.920838 20015 ts_tablet_manager.cc:1428] T f2b486dd27a34b3ea5ba4e265afe08de P b611e8f4f86d4d219aab4c221b1b0b37: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260924 05:02:46.920974 19733 heartbeater.cc:503] Master 127.0.98.62:42775 was elected leader, sending a full tablet report...
I20260924 05:02:46.921056 20016 ts_tablet_manager.cc:1428] T f2b486dd27a34b3ea5ba4e265afe08de P 07927f799ebf4bac807617b5404c7547: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 05:02:46.921082 19867 heartbeater.cc:503] Master 127.0.98.62:42775 was elected leader, sending a full tablet report...
W20260924 05:02:46.980000 19868 tablet.cc:2369] T f2b486dd27a34b3ea5ba4e265afe08de P 07927f799ebf4bac807617b5404c7547: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 05:02:47.083987 19734 tablet.cc:2369] T f2b486dd27a34b3ea5ba4e265afe08de P b611e8f4f86d4d219aab4c221b1b0b37: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 05:02:47.127204 20002 tablet.cc:2369] T f2b486dd27a34b3ea5ba4e265afe08de P 1a50d2ffa7fe40a5a4e3b5759ac910a8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 05:02:47.184283 20021 raft_consensus.cc:482] T f2b486dd27a34b3ea5ba4e265afe08de P 1a50d2ffa7fe40a5a4e3b5759ac910a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 05:02:47.184442 20021 raft_consensus.cc:504] T f2b486dd27a34b3ea5ba4e265afe08de P 1a50d2ffa7fe40a5a4e3b5759ac910a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a50d2ffa7fe40a5a4e3b5759ac910a8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40493 } } peers { permanent_uuid: "b611e8f4f86d4d219aab4c221b1b0b37" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37945 } } peers { permanent_uuid: "07927f799ebf4bac807617b5404c7547" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35961 } }
I20260924 05:02:47.184845 20021 leader_election.cc:288] T f2b486dd27a34b3ea5ba4e265afe08de P 1a50d2ffa7fe40a5a4e3b5759ac910a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b611e8f4f86d4d219aab4c221b1b0b37 (127.0.98.1:37945), 07927f799ebf4bac807617b5404c7547 (127.0.98.2:35961)
I20260924 05:02:47.187654 20014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:47.185140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54507 (local address 127.0.98.1:37945)
0924 05:02:47.185181 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 05:02:47.185183 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:02:47.185191 (+ 8us) server_negotiation.cc:407] Connection header received
0924 05:02:47.185223 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:02:47.185224 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:02:47.185235 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:02:47.185351 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:02:47.185511 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:47.186014 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:47.186995 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:47.187348 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:47.187398 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:47.187408 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 05:02:47.187549 (+ 141us) server_negotiation.cc:299] Negotiation successful
0924 05:02:47.187585 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 05:02:47.187652 20003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:47.185044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37945 (local address 127.0.98.3:54507)
0924 05:02:47.185091 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:47.185102 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 05:02:47.185146 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:02:47.185361 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:02:47.185361 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:02:47.185363 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:02:47.185456 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:47.185458 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:47.186087 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:47.186088 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:47.186901 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:47.186904 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:47.187415 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:47.187417 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:47.187491 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:47.187495 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 05:02:47.187532 (+ 37us) client_negotiation.cc:240] Negotiation successful
0924 05:02:47.187565 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26,"mutex_wait_us":34}
I20260924 05:02:47.187691 20011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:47.185095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40789 (local address 127.0.98.2:35961)
0924 05:02:47.185157 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 05:02:47.185159 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:02:47.185324 (+ 165us) server_negotiation.cc:407] Connection header received
0924 05:02:47.185385 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:02:47.185386 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:02:47.185399 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:02:47.185469 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:02:47.185608 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:47.186069 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:47.187003 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:47.187375 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:47.187440 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:47.187449 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 05:02:47.187594 (+ 145us) server_negotiation.cc:299] Negotiation successful
0924 05:02:47.187634 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 05:02:47.187747 20024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:47.185044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35961 (local address 127.0.98.3:40789)
0924 05:02:47.185249 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:47.185260 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 05:02:47.185328 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:02:47.185481 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:02:47.185482 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:02:47.185484 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:02:47.185551 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:47.185553 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:47.186144 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:47.186144 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:47.186907 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:47.186909 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:47.187456 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:47.187456 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:47.187522 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:47.187526 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 05:02:47.187576 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 05:02:47.187607 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"mutex_wait_us":54,"thread_start_us":97,"threads_started":1}
I20260924 05:02:47.188007 19687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2b486dd27a34b3ea5ba4e265afe08de" candidate_uuid: "1a50d2ffa7fe40a5a4e3b5759ac910a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b611e8f4f86d4d219aab4c221b1b0b37" is_pre_election: true
I20260924 05:02:47.188077 19821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2b486dd27a34b3ea5ba4e265afe08de" candidate_uuid: "1a50d2ffa7fe40a5a4e3b5759ac910a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07927f799ebf4bac807617b5404c7547" is_pre_election: true
I20260924 05:02:47.188189 19687 raft_consensus.cc:2454] T f2b486dd27a34b3ea5ba4e265afe08de P b611e8f4f86d4d219aab4c221b1b0b37 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a50d2ffa7fe40a5a4e3b5759ac910a8 in term 0.
I20260924 05:02:47.188189 19821 raft_consensus.cc:2454] T f2b486dd27a34b3ea5ba4e265afe08de P 07927f799ebf4bac807617b5404c7547 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a50d2ffa7fe40a5a4e3b5759ac910a8 in term 0.
I20260924 05:02:47.188387 19888 leader_election.cc:302] T f2b486dd27a34b3ea5ba4e265afe08de P 1a50d2ffa7fe40a5a4e3b5759ac910a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07927f799ebf4bac807617b5404c7547, 1a50d2ffa7fe40a5a4e3b5759ac910a8; no voters:
I20260924 05:02:47.188548 20021 raft_consensus.cc:2781] T f2b486dd27a34b3ea5ba4e265afe08de P 1a50d2ffa7fe40a5a4e3b5759ac910a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 05:02:47.188586 20021 raft_consensus.cc:482] T f2b486dd27a34b3ea5ba4e265afe08de P 1a50d2ffa7fe40a5a4e3b5759ac910a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 05:02:47.188601 20021 raft_consensus.cc:3037] T f2b486dd27a34b3ea5ba4e265afe08de P 1a50d2ffa7fe40a5a4e3b5759ac910a8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:02:47.189253 20021 raft_consensus.cc:504] T f2b486dd27a34b3ea5ba4e265afe08de P 1a50d2ffa7fe40a5a4e3b5759ac910a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a50d2ffa7fe40a5a4e3b5759ac910a8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40493 } } peers { permanent_uuid: "b611e8f4f86d4d219aab4c221b1b0b37" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37945 } } peers { permanent_uuid: "07927f799ebf4bac807617b5404c7547" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35961 } }
I20260924 05:02:47.189370 20021 leader_election.cc:288] T f2b486dd27a34b3ea5ba4e265afe08de P 1a50d2ffa7fe40a5a4e3b5759ac910a8 [CANDIDATE]: Term 1 election: Requested vote from peers b611e8f4f86d4d219aab4c221b1b0b37 (127.0.98.1:37945), 07927f799ebf4bac807617b5404c7547 (127.0.98.2:35961)
I20260924 05:02:47.189523 19821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2b486dd27a34b3ea5ba4e265afe08de" candidate_uuid: "1a50d2ffa7fe40a5a4e3b5759ac910a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07927f799ebf4bac807617b5404c7547"
I20260924 05:02:47.189558 19687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2b486dd27a34b3ea5ba4e265afe08de" candidate_uuid: "1a50d2ffa7fe40a5a4e3b5759ac910a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b611e8f4f86d4d219aab4c221b1b0b37"
I20260924 05:02:47.189602 19821 raft_consensus.cc:3037] T f2b486dd27a34b3ea5ba4e265afe08de P 07927f799ebf4bac807617b5404c7547 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:02:47.189618 19687 raft_consensus.cc:3037] T f2b486dd27a34b3ea5ba4e265afe08de P b611e8f4f86d4d219aab4c221b1b0b37 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:02:47.190423 19821 raft_consensus.cc:2454] T f2b486dd27a34b3ea5ba4e265afe08de P 07927f799ebf4bac807617b5404c7547 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a50d2ffa7fe40a5a4e3b5759ac910a8 in term 1.
I20260924 05:02:47.190423 19687 raft_consensus.cc:2454] T f2b486dd27a34b3ea5ba4e265afe08de P b611e8f4f86d4d219aab4c221b1b0b37 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a50d2ffa7fe40a5a4e3b5759ac910a8 in term 1.
I20260924 05:02:47.190606 19888 leader_election.cc:302] T f2b486dd27a34b3ea5ba4e265afe08de P 1a50d2ffa7fe40a5a4e3b5759ac910a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07927f799ebf4bac807617b5404c7547, 1a50d2ffa7fe40a5a4e3b5759ac910a8; no voters:
I20260924 05:02:47.190739 20021 raft_consensus.cc:2781] T f2b486dd27a34b3ea5ba4e265afe08de P 1a50d2ffa7fe40a5a4e3b5759ac910a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:02:47.190915 20021 raft_consensus.cc:686] T f2b486dd27a34b3ea5ba4e265afe08de P 1a50d2ffa7fe40a5a4e3b5759ac910a8 [term 1 LEADER]: Becoming Leader. State: Replica: 1a50d2ffa7fe40a5a4e3b5759ac910a8, State: Running, Role: LEADER
I20260924 05:02:47.191009 20021 consensus_queue.cc:237] T f2b486dd27a34b3ea5ba4e265afe08de P 1a50d2ffa7fe40a5a4e3b5759ac910a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a50d2ffa7fe40a5a4e3b5759ac910a8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40493 } } peers { permanent_uuid: "b611e8f4f86d4d219aab4c221b1b0b37" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37945 } } peers { permanent_uuid: "07927f799ebf4bac807617b5404c7547" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35961 } }
I20260924 05:02:47.191778 19542 catalog_manager.cc:5495] T f2b486dd27a34b3ea5ba4e265afe08de P 1a50d2ffa7fe40a5a4e3b5759ac910a8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1a50d2ffa7fe40a5a4e3b5759ac910a8 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "1a50d2ffa7fe40a5a4e3b5759ac910a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a50d2ffa7fe40a5a4e3b5759ac910a8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40493 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b611e8f4f86d4d219aab4c221b1b0b37" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07927f799ebf4bac807617b5404c7547" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35961 } health_report { overall_health: UNKNOWN } } }
I20260924 05:02:47.263231 20013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:47.260831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42900 (local address 127.0.98.3:40493)
0924 05:02:47.260887 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 05:02:47.260889 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:02:47.260898 (+ 9us) server_negotiation.cc:407] Connection header received
0924 05:02:47.260935 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:02:47.260936 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:02:47.260953 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:02:47.261016 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:02:47.261202 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:47.261707 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:47.262497 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:47.262676 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:47.262722 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:47.262905 (+ 183us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:02:47.263003 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:02:47.263058 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 05:02:47.263134 (+ 76us) server_negotiation.cc:299] Negotiation successful
0924 05:02:47.263151 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 05:02:47.264914 19821 raft_consensus.cc:1261] T f2b486dd27a34b3ea5ba4e265afe08de P 07927f799ebf4bac807617b5404c7547 [term 1 FOLLOWER]: Refusing update from remote peer 1a50d2ffa7fe40a5a4e3b5759ac910a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:02:47.264916 19687 raft_consensus.cc:1261] T f2b486dd27a34b3ea5ba4e265afe08de P b611e8f4f86d4d219aab4c221b1b0b37 [term 1 FOLLOWER]: Refusing update from remote peer 1a50d2ffa7fe40a5a4e3b5759ac910a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:02:47.265280 20021 consensus_queue.cc:1035] T f2b486dd27a34b3ea5ba4e265afe08de P 1a50d2ffa7fe40a5a4e3b5759ac910a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b611e8f4f86d4d219aab4c221b1b0b37" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:02:47.265441 20025 consensus_queue.cc:1035] T f2b486dd27a34b3ea5ba4e265afe08de P 1a50d2ffa7fe40a5a4e3b5759ac910a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07927f799ebf4bac807617b5404c7547" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:02:47.267606 20028 mvcc.cc:205] Tried to move back new op lower bound from 7332766381114466304 to 7332766380814913536. Current Snapshot: MvccSnapshot[applied={T|T < 7332766381114466304}]
I20260924 05:02:47.268121 20029 mvcc.cc:205] Tried to move back new op lower bound from 7332766381114466304 to 7332766380814913536. Current Snapshot: MvccSnapshot[applied={T|T < 7332766381114466304}]
I20260924 05:02:47.270413 19542 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43994:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:02:47.270512 19542 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43994:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:02:47.271250 19542 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b0d0a9b4d4064a9f9053b85fe1e59d07: Sending DeleteTablet for 3 replicas of tablet f2b486dd27a34b3ea5ba4e265afe08de
I20260924 05:02:47.271593 19935 tablet_service.cc:1514] Processing DeleteTablet for tablet f2b486dd27a34b3ea5ba4e265afe08de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:02:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42890
I20260924 05:02:47.271597 19801 tablet_service.cc:1514] Processing DeleteTablet for tablet f2b486dd27a34b3ea5ba4e265afe08de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:02:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54522
I20260924 05:02:47.271821 20038 tablet_replica.cc:330] T f2b486dd27a34b3ea5ba4e265afe08de P 1a50d2ffa7fe40a5a4e3b5759ac910a8: stopping tablet replica
I20260924 05:02:47.271831 19667 tablet_service.cc:1514] Processing DeleteTablet for tablet f2b486dd27a34b3ea5ba4e265afe08de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:02:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41372
I20260924 05:02:47.271912 20038 raft_consensus.cc:2229] T f2b486dd27a34b3ea5ba4e265afe08de P 1a50d2ffa7fe40a5a4e3b5759ac910a8 [term 1 LEADER]: Raft consensus shutting down.
I20260924 05:02:47.272025 20038 raft_consensus.cc:2258] T f2b486dd27a34b3ea5ba4e265afe08de P 1a50d2ffa7fe40a5a4e3b5759ac910a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:02:47.272071 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 19604
I20260924 05:02:47.272152 20040 tablet_replica.cc:330] T f2b486dd27a34b3ea5ba4e265afe08de P b611e8f4f86d4d219aab4c221b1b0b37: stopping tablet replica
I20260924 05:02:47.272228 20040 raft_consensus.cc:2229] T f2b486dd27a34b3ea5ba4e265afe08de P b611e8f4f86d4d219aab4c221b1b0b37 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:02:47.272312 20040 raft_consensus.cc:2258] T f2b486dd27a34b3ea5ba4e265afe08de P b611e8f4f86d4d219aab4c221b1b0b37 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:02:47.273938 20039 tablet_replica.cc:330] T f2b486dd27a34b3ea5ba4e265afe08de P 07927f799ebf4bac807617b5404c7547: stopping tablet replica
I20260924 05:02:47.274067 20039 raft_consensus.cc:2229] T f2b486dd27a34b3ea5ba4e265afe08de P 07927f799ebf4bac807617b5404c7547 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:02:47.274207 20039 raft_consensus.cc:2258] T f2b486dd27a34b3ea5ba4e265afe08de P 07927f799ebf4bac807617b5404c7547 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:02:47.274354 20038 ts_tablet_manager.cc:1905] T f2b486dd27a34b3ea5ba4e265afe08de P 1a50d2ffa7fe40a5a4e3b5759ac910a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:02:47.274681 20039 ts_tablet_manager.cc:1905] T f2b486dd27a34b3ea5ba4e265afe08de P 07927f799ebf4bac807617b5404c7547: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:02:47.276250 20038 ts_tablet_manager.cc:1918] T f2b486dd27a34b3ea5ba4e265afe08de P 1a50d2ffa7fe40a5a4e3b5759ac910a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 05:02:47.276250 20039 ts_tablet_manager.cc:1918] T f2b486dd27a34b3ea5ba4e265afe08de P 07927f799ebf4bac807617b5404c7547: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 05:02:47.276338 20039 log.cc:1196] T f2b486dd27a34b3ea5ba4e265afe08de P 07927f799ebf4bac807617b5404c7547: Deleting WAL directory at /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225929848825-392-0/minicluster-data/ts-1/wal/wals/f2b486dd27a34b3ea5ba4e265afe08de
I20260924 05:02:47.276432 20038 log.cc:1196] T f2b486dd27a34b3ea5ba4e265afe08de P 1a50d2ffa7fe40a5a4e3b5759ac910a8: Deleting WAL directory at /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790225929848825-392-0/minicluster-data/ts-2/wal/wals/f2b486dd27a34b3ea5ba4e265afe08de
I20260924 05:02:47.276638 20039 ts_tablet_manager.cc:1939] T f2b486dd27a34b3ea5ba4e265afe08de P 07927f799ebf4bac807617b5404c7547: Deleting consensus metadata
I20260924 05:02:47.276888 20038 ts_tablet_manager.cc:1939] T f2b486dd27a34b3ea5ba4e265afe08de P 1a50d2ffa7fe40a5a4e3b5759ac910a8: Deleting consensus metadata
I20260924 05:02:47.277176 19526 catalog_manager.cc:4841] TS 07927f799ebf4bac807617b5404c7547 (127.0.98.2:35961): tablet f2b486dd27a34b3ea5ba4e265afe08de (table test-table [id=fcc5bc78cb7541f180969b228a37c541]) successfully deleted
I20260924 05:02:47.277298 19529 catalog_manager.cc:4841] TS 1a50d2ffa7fe40a5a4e3b5759ac910a8 (127.0.98.3:40493): tablet f2b486dd27a34b3ea5ba4e265afe08de (table test-table [id=fcc5bc78cb7541f180969b228a37c541]) successfully deleted
W20260924 05:02:47.277846 19529 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37945 (error 108)
I20260924 05:02:47.278043 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 19738
I20260924 05:02:47.278559 20009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:47.278166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37945 (local address 127.0.0.1:41376)
0924 05:02:47.278486 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:47.278524 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":302}
W20260924 05:02:47.279100 19529 catalog_manager.cc:4584] TS b611e8f4f86d4d219aab4c221b1b0b37 (127.0.98.1:37945): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f2b486dd27a34b3ea5ba4e265afe08de: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37945: connect: Connection refused (error 111)
I20260924 05:02:47.283113 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 19872
I20260924 05:02:47.287149 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 19510
2026-09-24T05:02:47Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2065 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-24T05:02:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:02:47Z Disabled control of system clock
I20260924 05:02:47.312700 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225929848825-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225929848825-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225929848825-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38395
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:43139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38395
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 05:02:47.390017 20045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:02:47.390226 20045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:02:47.390245 20045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:02:47.391300 20045 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:02:47.391333 20045 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:02:47.391345 20045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:02:47.391355 20045 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:02:47.391366 20045 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:02:47.392603 20045 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225929848825-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225929848825-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38395
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20045
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225929848825-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:02:47.392782 20045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:02:47.393013 20045 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:02:47.397508 20051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:02:47.397528 20050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:02:47.397527 20045 server_base.cc:1034] running on GCE node
W20260924 05:02:47.397528 20053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:02:47.397943 20045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:02:47.398175 20045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:02:47.399320 20045 hybrid_clock.cc:648] HybridClock initialized: now 1790226167399282 us; error 52 us; skew 500 ppm
I20260924 05:02:47.400519 20045 webserver.cc:460] Webserver started at http://127.0.98.62:32985/ using document root <none> and password file <none>
I20260924 05:02:47.400725 20045 fs_manager.cc:359] Metadata directory not provided
I20260924 05:02:47.400770 20045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:02:47.400894 20045 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:02:47.401930 20045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225929848825-392-0/minicluster-data/master-0/data/instance:
uuid: "de72094977894f0a809738ff9fb66fd9"
format_stamp: "Formatted at 2026-09-24 05:02:47 on dist-test-slave-70st"
server_key: "bde3e9676985b0267a65bdfb987019f8"
server_key_iv: "b870416727004abda0fb25a513e3c80d"
server_key_version: "encryptionkey@0"
I20260924 05:02:47.402231 20045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225929848825-392-0/minicluster-data/master-0/wal/instance:
uuid: "de72094977894f0a809738ff9fb66fd9"
format_stamp: "Formatted at 2026-09-24 05:02:47 on dist-test-slave-70st"
server_key: "bde3e9676985b0267a65bdfb987019f8"
server_key_iv: "b870416727004abda0fb25a513e3c80d"
server_key_version: "encryptionkey@0"
I20260924 05:02:47.403430 20045 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 05:02:47.404167 20059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:02:47.404295 20045 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:02:47.404371 20045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225929848825-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225929848825-392-0/minicluster-data/master-0/wal
uuid: "de72094977894f0a809738ff9fb66fd9"
format_stamp: "Formatted at 2026-09-24 05:02:47 on dist-test-slave-70st"
server_key: "bde3e9676985b0267a65bdfb987019f8"
server_key_iv: "b870416727004abda0fb25a513e3c80d"
server_key_version: "encryptionkey@0"
I20260924 05:02:47.404436 20045 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225929848825-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225929848825-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225929848825-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:02:47.413654 20045 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:02:47.413947 20045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:02:47.414070 20045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:02:47.417945 20045 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38395
I20260924 05:02:47.417974 20111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38395 every 8 connection(s)
I20260924 05:02:47.418290 20045 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:02:47.419190 20112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:02:47.421386 20112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de72094977894f0a809738ff9fb66fd9: Bootstrap starting.
I20260924 05:02:47.421952 20112 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de72094977894f0a809738ff9fb66fd9: Neither blocks nor log segments found. Creating new log.
I20260924 05:02:47.422209 20112 log.cc:824] T 00000000000000000000000000000000 P de72094977894f0a809738ff9fb66fd9: Log is configured to *not* fsync() on all Append() calls
I20260924 05:02:47.422822 20112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de72094977894f0a809738ff9fb66fd9: No bootstrap required, opened a new log
I20260924 05:02:47.424366 20112 raft_consensus.cc:348] T 00000000000000000000000000000000 P de72094977894f0a809738ff9fb66fd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de72094977894f0a809738ff9fb66fd9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38395 } }
I20260924 05:02:47.424499 20112 raft_consensus.cc:374] T 00000000000000000000000000000000 P de72094977894f0a809738ff9fb66fd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:02:47.424521 20112 raft_consensus.cc:729] T 00000000000000000000000000000000 P de72094977894f0a809738ff9fb66fd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de72094977894f0a809738ff9fb66fd9, State: Initialized, Role: FOLLOWER
I20260924 05:02:47.424649 20112 consensus_queue.cc:260] T 00000000000000000000000000000000 P de72094977894f0a809738ff9fb66fd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de72094977894f0a809738ff9fb66fd9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38395 } }
I20260924 05:02:47.424710 20112 raft_consensus.cc:388] T 00000000000000000000000000000000 P de72094977894f0a809738ff9fb66fd9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:02:47.424731 20112 raft_consensus.cc:482] T 00000000000000000000000000000000 P de72094977894f0a809738ff9fb66fd9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:02:47.424774 20112 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de72094977894f0a809738ff9fb66fd9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:02:47.425244 20112 raft_consensus.cc:504] T 00000000000000000000000000000000 P de72094977894f0a809738ff9fb66fd9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de72094977894f0a809738ff9fb66fd9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38395 } }
I20260924 05:02:47.425335 20112 leader_election.cc:302] T 00000000000000000000000000000000 P de72094977894f0a809738ff9fb66fd9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de72094977894f0a809738ff9fb66fd9; no voters:
I20260924 05:02:47.425500 20112 leader_election.cc:288] T 00000000000000000000000000000000 P de72094977894f0a809738ff9fb66fd9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:02:47.425544 20115 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de72094977894f0a809738ff9fb66fd9 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:02:47.425690 20115 raft_consensus.cc:686] T 00000000000000000000000000000000 P de72094977894f0a809738ff9fb66fd9 [term 1 LEADER]: Becoming Leader. State: Replica: de72094977894f0a809738ff9fb66fd9, State: Running, Role: LEADER
I20260924 05:02:47.425755 20112 sys_catalog.cc:564] T 00000000000000000000000000000000 P de72094977894f0a809738ff9fb66fd9 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:02:47.425806 20115 consensus_queue.cc:237] T 00000000000000000000000000000000 P de72094977894f0a809738ff9fb66fd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de72094977894f0a809738ff9fb66fd9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38395 } }
I20260924 05:02:47.426172 20117 sys_catalog.cc:455] T 00000000000000000000000000000000 P de72094977894f0a809738ff9fb66fd9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader de72094977894f0a809738ff9fb66fd9. Latest consensus state: current_term: 1 leader_uuid: "de72094977894f0a809738ff9fb66fd9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de72094977894f0a809738ff9fb66fd9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38395 } } }
I20260924 05:02:47.426270 20116 sys_catalog.cc:455] T 00000000000000000000000000000000 P de72094977894f0a809738ff9fb66fd9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "de72094977894f0a809738ff9fb66fd9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de72094977894f0a809738ff9fb66fd9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38395 } } }
I20260924 05:02:47.426332 20117 sys_catalog.cc:458] T 00000000000000000000000000000000 P de72094977894f0a809738ff9fb66fd9 [sys.catalog]: This master's current role is: LEADER
I20260924 05:02:47.426401 20116 sys_catalog.cc:458] T 00000000000000000000000000000000 P de72094977894f0a809738ff9fb66fd9 [sys.catalog]: This master's current role is: LEADER
I20260924 05:02:47.426720 20124 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:02:47.426756 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 20045
I20260924 05:02:47.426841 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225929848825-392-0/minicluster-data/master-0/wal/instance
I20260924 05:02:47.427042 392 external_mini_cluster.cc:1428] Setting key 97c9c34d43af9a0c504f97d1b25a33d2
I20260924 05:02:47.427235 20124 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:02:47.429785 20124 catalog_manager.cc:1347] Generated new cluster ID: a3faf4a5788a48fe9736c6a9ce2a567e
I20260924 05:02:47.429842 20124 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:02:47.432503 20134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:47.427505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44968 (local address 127.0.98.62:38395)
0924 05:02:47.427889 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0924 05:02:47.427899 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 05:02:47.427929 (+ 30us) server_negotiation.cc:407] Connection header received
0924 05:02:47.428041 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:02:47.428047 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:02:47.428116 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:02:47.428393 (+ 277us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:02:47.428486 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:47.429343 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:47.430405 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:47.431049 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:47.431173 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:47.431807 (+ 634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:02:47.431811 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:02:47.431811 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:02:47.432014 (+ 203us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:02:47.432064 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:02:47.432080 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:02:47.432149 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 05:02:47.432206 (+ 57us) server_negotiation.cc:299] Negotiation successful
0924 05:02:47.432284 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":84,"threads_started":1}
I20260924 05:02:47.446743 20124 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:02:47.447288 20124 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:02:47.451748 20124 catalog_manager.cc:5868] T 00000000000000000000000000000000 P de72094977894f0a809738ff9fb66fd9: Generated new TSK 0
I20260924 05:02:47.451930 20124 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 05:02:47.452986 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225929848825-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225929848825-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225929848825-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225929848825-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38395
--builtin_ntp_servers=127.0.98.20:43139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 05:02:47.531378 20136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:02:47.531576 20136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:02:47.531606 20136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:02:47.532737 20136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:02:47.532774 20136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:02:47.532807 20136 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 05:02:47.534175 20136 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225929848825-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225929848825-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225929848825-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20136
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225929848825-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:02:47.534392 20136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:02:47.534646 20136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:02:47.539381 20142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:02:47.539392 20141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:02:47.539495 20144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:02:47.539700 20136 server_base.cc:1034] running on GCE node
I20260924 05:02:47.539920 20136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:02:47.540184 20136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:02:47.541360 20136 hybrid_clock.cc:648] HybridClock initialized: now 1790226167541338 us; error 36 us; skew 500 ppm
I20260924 05:02:47.542699 20136 webserver.cc:460] Webserver started at http://127.0.98.1:42341/ using document root <none> and password file <none>
I20260924 05:02:47.542937 20136 fs_manager.cc:359] Metadata directory not provided
I20260924 05:02:47.542994 20136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:02:47.543107 20136 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:02:47.544440 20136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225929848825-392-0/minicluster-data/ts-0/data/instance:
uuid: "26540b3808bf4f47805c2120efabc138"
format_stamp: "Formatted at 2026-09-24 05:02:47 on dist-test-slave-70st"
server_key: "3dcdf125b9cbb2434af73c0b9b2912e4"
server_key_iv: "5baa5e1bee257418b2b5b91fa085f6be"
server_key_version: "encryptionkey@0"
I20260924 05:02:47.544795 20136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225929848825-392-0/minicluster-data/ts-0/wal/instance:
uuid: "26540b3808bf4f47805c2120efabc138"
format_stamp: "Formatted at 2026-09-24 05:02:47 on dist-test-slave-70st"
server_key: "3dcdf125b9cbb2434af73c0b9b2912e4"
server_key_iv: "5baa5e1bee257418b2b5b91fa085f6be"
server_key_version: "encryptionkey@0"
I20260924 05:02:47.546415 20136 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 05:02:47.547258 20150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:02:47.547432 20136 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 05:02:47.547518 20136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225929848825-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225929848825-392-0/minicluster-data/ts-0/wal
uuid: "26540b3808bf4f47805c2120efabc138"
format_stamp: "Formatted at 2026-09-24 05:02:47 on dist-test-slave-70st"
server_key: "3dcdf125b9cbb2434af73c0b9b2912e4"
server_key_iv: "5baa5e1bee257418b2b5b91fa085f6be"
server_key_version: "encryptionkey@0"
I20260924 05:02:47.547588 20136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225929848825-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225929848825-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225929848825-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:02:47.562220 20136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:02:47.562556 20136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:02:47.562667 20136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:02:47.562879 20136 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:02:47.563159 20136 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:02:47.563189 20136 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:02:47.563213 20136 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:02:47.563230 20136 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:02:47.568786 20136 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46865
I20260924 05:02:47.568863 20263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46865 every 8 connection(s)
I20260924 05:02:47.569185 20136 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225929848825-392-0/minicluster-data/ts-0/data/info.pb
I20260924 05:02:47.574694 20134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:47.570018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34727 (local address 127.0.98.62:38395)
0924 05:02:47.570072 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 05:02:47.570074 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:02:47.570242 (+ 168us) server_negotiation.cc:407] Connection header received
0924 05:02:47.570755 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:02:47.570756 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:02:47.570781 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:02:47.570849 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:02:47.571585 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:47.572113 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:47.573144 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:47.573350 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:47.573438 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:47.574394 (+ 956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:02:47.574395 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:02:47.574396 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:02:47.574404 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:02:47.574419 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:02:47.574423 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:02:47.574480 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 05:02:47.574600 (+ 120us) server_negotiation.cc:299] Negotiation successful
0924 05:02:47.574629 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 05:02:47.574898 20266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:47.569937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38395 (local address 127.0.98.1:34727)
0924 05:02:47.570118 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:47.570160 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 05:02:47.570645 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:02:47.570954 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:02:47.570958 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:02:47.570968 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:02:47.571461 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:47.571465 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:47.572195 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:47.572196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:47.573055 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:47.573058 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:47.573483 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:47.573485 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:47.573618 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:47.574235 (+ 617us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 05:02:47.574236 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:02:47.574267 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 05:02:47.574292 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:02:47.574516 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:02:47.574523 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:02:47.574526 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:02:47.574600 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 05:02:47.574690 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":66,"threads_started":1}
I20260924 05:02:47.575555 20264 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38395
I20260924 05:02:47.575649 20264 heartbeater.cc:464] Registering TS with master...
I20260924 05:02:47.575804 20264 heartbeater.cc:511] Master 127.0.98.62:38395 requested a full tablet report, sending...
I20260924 05:02:47.576285 20076 ts_manager.cc:194] Registered new tserver with Master: 26540b3808bf4f47805c2120efabc138 (127.0.98.1:46865)
I20260924 05:02:47.576887 20076 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:34727
I20260924 05:02:47.577633 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 20136
I20260924 05:02:47.577728 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225929848825-392-0/minicluster-data/ts-0/wal/instance
I20260924 05:02:47.577884 392 external_mini_cluster.cc:1428] Setting key 17e7db0f93e1986960dd1621b10338ce
I20260924 05:02:47.578676 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225929848825-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225929848825-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225929848825-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225929848825-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38395
--builtin_ntp_servers=127.0.98.20:43139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 05:02:47.659358 20267 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:02:47.659566 20267 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:02:47.659586 20267 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:02:47.660717 20267 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:02:47.660756 20267 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:02:47.660780 20267 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 05:02:47.662106 20267 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225929848825-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225929848825-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225929848825-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20267
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225929848825-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:02:47.662292 20267 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:02:47.662539 20267 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:02:47.667447 20273 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:02:47.667454 20275 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:02:47.667455 20272 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:02:47.667687 20267 server_base.cc:1034] running on GCE node
I20260924 05:02:47.667891 20267 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:02:47.668123 20267 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:02:47.669272 20267 hybrid_clock.cc:648] HybridClock initialized: now 1790226167669252 us; error 36 us; skew 500 ppm
I20260924 05:02:47.670506 20267 webserver.cc:460] Webserver started at http://127.0.98.2:36051/ using document root <none> and password file <none>
I20260924 05:02:47.670738 20267 fs_manager.cc:359] Metadata directory not provided
I20260924 05:02:47.670782 20267 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:02:47.670928 20267 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:02:47.671958 20267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225929848825-392-0/minicluster-data/ts-1/data/instance:
uuid: "3ffb11b801484c168d4ae90c461d4fc4"
format_stamp: "Formatted at 2026-09-24 05:02:47 on dist-test-slave-70st"
server_key: "f97a1a079297764b8c33fac1382019e1"
server_key_iv: "54ce0ba525ce09f406687a1743f8ff2f"
server_key_version: "encryptionkey@0"
I20260924 05:02:47.672282 20267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225929848825-392-0/minicluster-data/ts-1/wal/instance:
uuid: "3ffb11b801484c168d4ae90c461d4fc4"
format_stamp: "Formatted at 2026-09-24 05:02:47 on dist-test-slave-70st"
server_key: "f97a1a079297764b8c33fac1382019e1"
server_key_iv: "54ce0ba525ce09f406687a1743f8ff2f"
server_key_version: "encryptionkey@0"
I20260924 05:02:47.673497 20267 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 05:02:47.674160 20281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:02:47.674288 20267 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:02:47.674369 20267 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225929848825-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225929848825-392-0/minicluster-data/ts-1/wal
uuid: "3ffb11b801484c168d4ae90c461d4fc4"
format_stamp: "Formatted at 2026-09-24 05:02:47 on dist-test-slave-70st"
server_key: "f97a1a079297764b8c33fac1382019e1"
server_key_iv: "54ce0ba525ce09f406687a1743f8ff2f"
server_key_version: "encryptionkey@0"
I20260924 05:02:47.674435 20267 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225929848825-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225929848825-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225929848825-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:02:47.700429 20267 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:02:47.700750 20267 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:02:47.700892 20267 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:02:47.701115 20267 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:02:47.701457 20267 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:02:47.701491 20267 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:02:47.701521 20267 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:02:47.701538 20267 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:02:47.706981 20267 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34691
I20260924 05:02:47.707038 20394 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34691 every 8 connection(s)
I20260924 05:02:47.707342 20267 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225929848825-392-0/minicluster-data/ts-1/data/info.pb
I20260924 05:02:47.712394 20134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:47.708033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35881 (local address 127.0.98.62:38395)
0924 05:02:47.708082 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 05:02:47.708084 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:02:47.708336 (+ 252us) server_negotiation.cc:407] Connection header received
0924 05:02:47.708839 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:02:47.708840 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:02:47.708851 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:02:47.708901 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:02:47.709865 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:47.710397 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:47.711319 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:47.711491 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:47.711533 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:47.712198 (+ 665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:02:47.712199 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:02:47.712199 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:02:47.712204 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:02:47.712218 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:02:47.712222 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:02:47.712259 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0924 05:02:47.712306 (+ 47us) server_negotiation.cc:299] Negotiation successful
0924 05:02:47.712333 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 05:02:47.712594 20397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:47.708026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38395 (local address 127.0.98.2:35881)
0924 05:02:47.708212 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:47.708246 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 05:02:47.708749 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:02:47.708975 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:02:47.708978 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:02:47.708989 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:02:47.709736 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:47.709744 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:47.710474 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:47.710475 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:47.711255 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:47.711256 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:47.711543 (+ 287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:47.711544 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:47.711648 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:47.712080 (+ 432us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 05:02:47.712080 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:02:47.712108 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 05:02:47.712132 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:02:47.712259 (+ 127us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:02:47.712262 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:02:47.712264 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 05:02:47.712308 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 05:02:47.712409 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":87,"threads_started":1}
I20260924 05:02:47.713056 20395 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38395
I20260924 05:02:47.713140 20395 heartbeater.cc:464] Registering TS with master...
I20260924 05:02:47.713316 20395 heartbeater.cc:511] Master 127.0.98.62:38395 requested a full tablet report, sending...
I20260924 05:02:47.713449 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 20267
I20260924 05:02:47.713554 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225929848825-392-0/minicluster-data/ts-1/wal/instance
I20260924 05:02:47.713685 20076 ts_manager.cc:194] Registered new tserver with Master: 3ffb11b801484c168d4ae90c461d4fc4 (127.0.98.2:34691)
I20260924 05:02:47.713794 392 external_mini_cluster.cc:1428] Setting key d350302db8bd5c61a619d0eb120a33cb
I20260924 05:02:47.714123 20076 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:35881
I20260924 05:02:47.714895 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225929848825-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225929848825-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225929848825-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225929848825-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38395
--builtin_ntp_servers=127.0.98.20:43139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 05:02:47.793175 20398 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:02:47.793385 20398 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:02:47.793404 20398 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:02:47.794561 20398 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:02:47.794598 20398 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:02:47.794621 20398 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 05:02:47.795945 20398 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225929848825-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225929848825-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225929848825-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20398
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225929848825-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:02:47.796157 20398 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:02:47.796423 20398 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 05:02:47.801189 20398 server_base.cc:1034] running on GCE node
W20260924 05:02:47.801172 20404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:02:47.801158 20403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:02:47.801173 20406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:02:47.801635 20398 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:02:47.801888 20398 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:02:47.803032 20398 hybrid_clock.cc:648] HybridClock initialized: now 1790226167803019 us; error 29 us; skew 500 ppm
I20260924 05:02:47.804281 20398 webserver.cc:460] Webserver started at http://127.0.98.3:40433/ using document root <none> and password file <none>
I20260924 05:02:47.804495 20398 fs_manager.cc:359] Metadata directory not provided
I20260924 05:02:47.804543 20398 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:02:47.804644 20398 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:02:47.805763 20398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225929848825-392-0/minicluster-data/ts-2/data/instance:
uuid: "9bc4920b33044157b66dab4f172e634a"
format_stamp: "Formatted at 2026-09-24 05:02:47 on dist-test-slave-70st"
server_key: "01ec6b501751e5d4be397a5e479b4805"
server_key_iv: "23f5c81dde8b838b5e2e17a38959b226"
server_key_version: "encryptionkey@0"
I20260924 05:02:47.806104 20398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225929848825-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9bc4920b33044157b66dab4f172e634a"
format_stamp: "Formatted at 2026-09-24 05:02:47 on dist-test-slave-70st"
server_key: "01ec6b501751e5d4be397a5e479b4805"
server_key_iv: "23f5c81dde8b838b5e2e17a38959b226"
server_key_version: "encryptionkey@0"
I20260924 05:02:47.807389 20398 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 05:02:47.808017 20412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:02:47.808180 20398 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:02:47.808261 20398 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225929848825-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225929848825-392-0/minicluster-data/ts-2/wal
uuid: "9bc4920b33044157b66dab4f172e634a"
format_stamp: "Formatted at 2026-09-24 05:02:47 on dist-test-slave-70st"
server_key: "01ec6b501751e5d4be397a5e479b4805"
server_key_iv: "23f5c81dde8b838b5e2e17a38959b226"
server_key_version: "encryptionkey@0"
I20260924 05:02:47.808320 20398 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225929848825-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225929848825-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225929848825-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:02:47.822580 20398 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:02:47.822896 20398 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:02:47.823026 20398 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:02:47.823233 20398 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 05:02:47.823553 20398 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 05:02:47.823586 20398 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 05:02:47.823616 20398 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 05:02:47.823632 20398 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 05:02:47.829103 20398 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39713
I20260924 05:02:47.829166 20525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39713 every 8 connection(s)
I20260924 05:02:47.829499 20398 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225929848825-392-0/minicluster-data/ts-2/data/info.pb
I20260924 05:02:47.834651 20134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:47.830330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53773 (local address 127.0.98.62:38395)
0924 05:02:47.830392 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 05:02:47.830394 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:02:47.830631 (+ 237us) server_negotiation.cc:407] Connection header received
0924 05:02:47.831126 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:02:47.831127 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:02:47.831138 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:02:47.831177 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:02:47.831861 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:47.832372 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:47.833351 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:47.833574 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:47.833637 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:47.834437 (+ 800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:02:47.834438 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:02:47.834438 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:02:47.834443 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:02:47.834456 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:02:47.834460 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:02:47.834500 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0924 05:02:47.834568 (+ 68us) server_negotiation.cc:299] Negotiation successful
0924 05:02:47.834590 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 05:02:47.834857 20528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:47.830301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38395 (local address 127.0.98.3:53773)
0924 05:02:47.830518 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:47.830552 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 05:02:47.831034 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:02:47.831243 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:02:47.831247 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:02:47.831256 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:02:47.831740 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:47.831744 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:47.832458 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:47.832459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:47.833257 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:47.833259 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:47.833655 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:47.833656 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:47.833777 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:47.834293 (+ 516us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 05:02:47.834294 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:02:47.834326 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 05:02:47.834350 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:02:47.834512 (+ 162us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:02:47.834514 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:02:47.834517 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:02:47.834566 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 05:02:47.834643 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":93,"threads_started":1}
I20260924 05:02:47.835316 20526 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38395
I20260924 05:02:47.835402 20526 heartbeater.cc:464] Registering TS with master...
I20260924 05:02:47.835552 20526 heartbeater.cc:511] Master 127.0.98.62:38395 requested a full tablet report, sending...
I20260924 05:02:47.835935 20075 ts_manager.cc:194] Registered new tserver with Master: 9bc4920b33044157b66dab4f172e634a (127.0.98.3:39713)
I20260924 05:02:47.836282 20075 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:53773
I20260924 05:02:47.839251 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 20398
I20260924 05:02:47.839351 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225929848825-392-0/minicluster-data/ts-2/wal/instance
I20260924 05:02:47.839519 392 external_mini_cluster.cc:1428] Setting key 2bc6417a3d7bcffe941350746db1622f
I20260924 05:02:47.840473 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 05:02:47.844280 20134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:47.841824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44970 (local address 127.0.98.62:38395)
0924 05:02:47.841879 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 05:02:47.841881 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:02:47.841960 (+ 79us) server_negotiation.cc:407] Connection header received
0924 05:02:47.842011 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:02:47.842012 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:02:47.842022 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:02:47.842064 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:02:47.842201 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:47.842648 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:47.843355 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:47.843526 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:47.843571 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:47.844093 (+ 522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:02:47.844094 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:02:47.844094 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:02:47.844098 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:02:47.844110 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:02:47.844114 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:02:47.844151 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0924 05:02:47.844187 (+ 36us) server_negotiation.cc:299] Negotiation successful
0924 05:02:47.844206 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 05:02:47.845623 20075 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44970:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 05:02:47.847066 20075 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 05:02:47.853040 20536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:47.849035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46865 (local address 127.0.0.1:34720)
0924 05:02:47.849369 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:47.849378 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:02:47.849457 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:02:47.849663 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:02:47.849664 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:02:47.849665 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:02:47.849731 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:47.849734 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:47.850774 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:47.850775 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:47.852032 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:47.852034 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:47.852763 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:47.852764 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:47.852837 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:47.852851 (+ 14us) client_negotiation.cc:769] Sending connection context
0924 05:02:47.852912 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 05:02:47.852963 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"mutex_wait_us":9,"thread_start_us":47,"threads_started":1}
I20260924 05:02:47.853050 20538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:47.849256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34720 (local address 127.0.98.1:46865)
0924 05:02:47.849443 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 05:02:47.849447 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 05:02:47.849458 (+ 11us) server_negotiation.cc:407] Connection header received
0924 05:02:47.849540 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:02:47.849543 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:02:47.849559 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:02:47.849629 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:02:47.849889 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:47.850708 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:47.852136 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:47.852680 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:47.852740 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:47.852831 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 05:02:47.852934 (+ 103us) server_negotiation.cc:299] Negotiation successful
0924 05:02:47.852975 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":65,"threads_started":1}
I20260924 05:02:47.853144 20534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:47.849040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39713 (local address 127.0.0.1:35218)
0924 05:02:47.849502 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:47.849511 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:02:47.849564 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:02:47.849763 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:02:47.849764 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:02:47.849766 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:02:47.849808 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:47.849810 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:47.851064 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:47.851065 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:47.852019 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:47.852021 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:47.852919 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:47.852920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:47.852978 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:47.852983 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 05:02:47.853032 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 05:02:47.853070 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"mutex_wait_us":25,"thread_start_us":99,"threads_started":1}
I20260924 05:02:47.853374 20535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:47.849035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34691 (local address 127.0.0.1:35166)
0924 05:02:47.849191 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:47.849226 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 05:02:47.849328 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:02:47.849656 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:02:47.849659 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:02:47.849661 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:02:47.849731 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:47.849733 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:47.851071 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:47.851072 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:47.852242 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:47.852244 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:47.853157 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:47.853158 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:47.853213 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:47.853216 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:02:47.853267 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 05:02:47.853300 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":31,"thread_start_us":94,"threads_started":1}
I20260924 05:02:47.853397 20539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:47.849259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35218 (local address 127.0.98.3:39713)
0924 05:02:47.849411 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 05:02:47.849443 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0924 05:02:47.849560 (+ 117us) server_negotiation.cc:407] Connection header received
0924 05:02:47.849623 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:02:47.849625 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:02:47.849638 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:02:47.849705 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:02:47.849887 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:47.850973 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:47.852090 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:47.852723 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:47.852787 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:47.852883 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 05:02:47.853281 (+ 398us) server_negotiation.cc:299] Negotiation successful
0924 05:02:47.853322 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":64,"threads_started":1}
I20260924 05:02:47.853612 20537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:47.849118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35166 (local address 127.0.98.2:34691)
0924 05:02:47.849288 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 05:02:47.849293 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 05:02:47.849319 (+ 26us) server_negotiation.cc:407] Connection header received
0924 05:02:47.849490 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:02:47.849493 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:02:47.849510 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:02:47.849580 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:02:47.849837 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:47.850973 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:47.852325 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:47.852998 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:47.853053 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:47.853126 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 05:02:47.853495 (+ 369us) server_negotiation.cc:299] Negotiation successful
0924 05:02:47.853539 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":72,"threads_started":1}
I20260924 05:02:47.854184 20329 tablet_service.cc:1467] Processing CreateTablet for tablet c3dc383b488545d983dff1095f57f894 (DEFAULT_TABLE table=test-table [id=97be90e8ecfc4bc988022a59cb51874d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:02:47.854547 20329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3dc383b488545d983dff1095f57f894. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:02:47.854516 20198 tablet_service.cc:1467] Processing CreateTablet for tablet c3dc383b488545d983dff1095f57f894 (DEFAULT_TABLE table=test-table [id=97be90e8ecfc4bc988022a59cb51874d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:02:47.854827 20198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3dc383b488545d983dff1095f57f894. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:02:47.857560 20460 tablet_service.cc:1467] Processing CreateTablet for tablet c3dc383b488545d983dff1095f57f894 (DEFAULT_TABLE table=test-table [id=97be90e8ecfc4bc988022a59cb51874d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 05:02:47.857889 20460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3dc383b488545d983dff1095f57f894. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:02:47.857944 20540 tablet_bootstrap.cc:492] T c3dc383b488545d983dff1095f57f894 P 26540b3808bf4f47805c2120efabc138: Bootstrap starting.
I20260924 05:02:47.858502 20540 tablet_bootstrap.cc:654] T c3dc383b488545d983dff1095f57f894 P 26540b3808bf4f47805c2120efabc138: Neither blocks nor log segments found. Creating new log.
I20260924 05:02:47.858759 20540 log.cc:824] T c3dc383b488545d983dff1095f57f894 P 26540b3808bf4f47805c2120efabc138: Log is configured to *not* fsync() on all Append() calls
I20260924 05:02:47.859062 20541 tablet_bootstrap.cc:492] T c3dc383b488545d983dff1095f57f894 P 3ffb11b801484c168d4ae90c461d4fc4: Bootstrap starting.
I20260924 05:02:47.859450 20540 tablet_bootstrap.cc:492] T c3dc383b488545d983dff1095f57f894 P 26540b3808bf4f47805c2120efabc138: No bootstrap required, opened a new log
I20260924 05:02:47.859517 20540 ts_tablet_manager.cc:1397] T c3dc383b488545d983dff1095f57f894 P 26540b3808bf4f47805c2120efabc138: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 05:02:47.859700 20541 tablet_bootstrap.cc:654] T c3dc383b488545d983dff1095f57f894 P 3ffb11b801484c168d4ae90c461d4fc4: Neither blocks nor log segments found. Creating new log.
I20260924 05:02:47.859992 20541 log.cc:824] T c3dc383b488545d983dff1095f57f894 P 3ffb11b801484c168d4ae90c461d4fc4: Log is configured to *not* fsync() on all Append() calls
I20260924 05:02:47.860651 20541 tablet_bootstrap.cc:492] T c3dc383b488545d983dff1095f57f894 P 3ffb11b801484c168d4ae90c461d4fc4: No bootstrap required, opened a new log
I20260924 05:02:47.860720 20541 ts_tablet_manager.cc:1397] T c3dc383b488545d983dff1095f57f894 P 3ffb11b801484c168d4ae90c461d4fc4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 05:02:47.861564 20544 tablet_bootstrap.cc:492] T c3dc383b488545d983dff1095f57f894 P 9bc4920b33044157b66dab4f172e634a: Bootstrap starting.
I20260924 05:02:47.862046 20544 tablet_bootstrap.cc:654] T c3dc383b488545d983dff1095f57f894 P 9bc4920b33044157b66dab4f172e634a: Neither blocks nor log segments found. Creating new log.
I20260924 05:02:47.862004 20540 raft_consensus.cc:348] T c3dc383b488545d983dff1095f57f894 P 26540b3808bf4f47805c2120efabc138 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ffb11b801484c168d4ae90c461d4fc4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34691 } } peers { permanent_uuid: "9bc4920b33044157b66dab4f172e634a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39713 } } peers { permanent_uuid: "26540b3808bf4f47805c2120efabc138" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46865 } }
I20260924 05:02:47.862176 20540 raft_consensus.cc:374] T c3dc383b488545d983dff1095f57f894 P 26540b3808bf4f47805c2120efabc138 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:02:47.862219 20540 raft_consensus.cc:729] T c3dc383b488545d983dff1095f57f894 P 26540b3808bf4f47805c2120efabc138 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26540b3808bf4f47805c2120efabc138, State: Initialized, Role: FOLLOWER
I20260924 05:02:47.862334 20544 log.cc:824] T c3dc383b488545d983dff1095f57f894 P 9bc4920b33044157b66dab4f172e634a: Log is configured to *not* fsync() on all Append() calls
I20260924 05:02:47.862352 20540 consensus_queue.cc:260] T c3dc383b488545d983dff1095f57f894 P 26540b3808bf4f47805c2120efabc138 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ffb11b801484c168d4ae90c461d4fc4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34691 } } peers { permanent_uuid: "9bc4920b33044157b66dab4f172e634a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39713 } } peers { permanent_uuid: "26540b3808bf4f47805c2120efabc138" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46865 } }
I20260924 05:02:47.862614 20540 ts_tablet_manager.cc:1428] T c3dc383b488545d983dff1095f57f894 P 26540b3808bf4f47805c2120efabc138: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 05:02:47.862682 20264 heartbeater.cc:503] Master 127.0.98.62:38395 was elected leader, sending a full tablet report...
I20260924 05:02:47.863214 20544 tablet_bootstrap.cc:492] T c3dc383b488545d983dff1095f57f894 P 9bc4920b33044157b66dab4f172e634a: No bootstrap required, opened a new log
I20260924 05:02:47.863291 20544 ts_tablet_manager.cc:1397] T c3dc383b488545d983dff1095f57f894 P 9bc4920b33044157b66dab4f172e634a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 05:02:47.863178 20541 raft_consensus.cc:348] T c3dc383b488545d983dff1095f57f894 P 3ffb11b801484c168d4ae90c461d4fc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ffb11b801484c168d4ae90c461d4fc4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34691 } } peers { permanent_uuid: "9bc4920b33044157b66dab4f172e634a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39713 } } peers { permanent_uuid: "26540b3808bf4f47805c2120efabc138" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46865 } }
I20260924 05:02:47.863359 20541 raft_consensus.cc:374] T c3dc383b488545d983dff1095f57f894 P 3ffb11b801484c168d4ae90c461d4fc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:02:47.863400 20541 raft_consensus.cc:729] T c3dc383b488545d983dff1095f57f894 P 3ffb11b801484c168d4ae90c461d4fc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ffb11b801484c168d4ae90c461d4fc4, State: Initialized, Role: FOLLOWER
I20260924 05:02:47.863513 20541 consensus_queue.cc:260] T c3dc383b488545d983dff1095f57f894 P 3ffb11b801484c168d4ae90c461d4fc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ffb11b801484c168d4ae90c461d4fc4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34691 } } peers { permanent_uuid: "9bc4920b33044157b66dab4f172e634a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39713 } } peers { permanent_uuid: "26540b3808bf4f47805c2120efabc138" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46865 } }
I20260924 05:02:47.863756 20541 ts_tablet_manager.cc:1428] T c3dc383b488545d983dff1095f57f894 P 3ffb11b801484c168d4ae90c461d4fc4: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 05:02:47.863790 20395 heartbeater.cc:503] Master 127.0.98.62:38395 was elected leader, sending a full tablet report...
I20260924 05:02:47.865396 20544 raft_consensus.cc:348] T c3dc383b488545d983dff1095f57f894 P 9bc4920b33044157b66dab4f172e634a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ffb11b801484c168d4ae90c461d4fc4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34691 } } peers { permanent_uuid: "9bc4920b33044157b66dab4f172e634a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39713 } } peers { permanent_uuid: "26540b3808bf4f47805c2120efabc138" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46865 } }
I20260924 05:02:47.865561 20544 raft_consensus.cc:374] T c3dc383b488545d983dff1095f57f894 P 9bc4920b33044157b66dab4f172e634a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:02:47.865585 20544 raft_consensus.cc:729] T c3dc383b488545d983dff1095f57f894 P 9bc4920b33044157b66dab4f172e634a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc4920b33044157b66dab4f172e634a, State: Initialized, Role: FOLLOWER
I20260924 05:02:47.865692 20544 consensus_queue.cc:260] T c3dc383b488545d983dff1095f57f894 P 9bc4920b33044157b66dab4f172e634a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ffb11b801484c168d4ae90c461d4fc4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34691 } } peers { permanent_uuid: "9bc4920b33044157b66dab4f172e634a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39713 } } peers { permanent_uuid: "26540b3808bf4f47805c2120efabc138" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46865 } }
I20260924 05:02:47.865911 20544 ts_tablet_manager.cc:1428] T c3dc383b488545d983dff1095f57f894 P 9bc4920b33044157b66dab4f172e634a: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260924 05:02:47.865955 20526 heartbeater.cc:503] Master 127.0.98.62:38395 was elected leader, sending a full tablet report...
W20260924 05:02:47.957862 20396 tablet.cc:2369] T c3dc383b488545d983dff1095f57f894 P 3ffb11b801484c168d4ae90c461d4fc4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 05:02:48.013974 20545 raft_consensus.cc:482] T c3dc383b488545d983dff1095f57f894 P 26540b3808bf4f47805c2120efabc138 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 05:02:48.014122 20545 raft_consensus.cc:504] T c3dc383b488545d983dff1095f57f894 P 26540b3808bf4f47805c2120efabc138 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ffb11b801484c168d4ae90c461d4fc4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34691 } } peers { permanent_uuid: "9bc4920b33044157b66dab4f172e634a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39713 } } peers { permanent_uuid: "26540b3808bf4f47805c2120efabc138" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46865 } }
I20260924 05:02:48.014508 20545 leader_election.cc:288] T c3dc383b488545d983dff1095f57f894 P 26540b3808bf4f47805c2120efabc138 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3ffb11b801484c168d4ae90c461d4fc4 (127.0.98.2:34691), 9bc4920b33044157b66dab4f172e634a (127.0.98.3:39713)
I20260924 05:02:48.017490 20537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:48.014841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51435 (local address 127.0.98.2:34691)
0924 05:02:48.014887 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 05:02:48.014889 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:02:48.015055 (+ 166us) server_negotiation.cc:407] Connection header received
0924 05:02:48.015105 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:02:48.015106 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:02:48.015116 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:02:48.015182 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:02:48.015422 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:48.015903 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:48.016796 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:48.017149 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:48.017207 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:48.017217 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 05:02:48.017368 (+ 151us) server_negotiation.cc:299] Negotiation successful
0924 05:02:48.017406 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 05:02:48.017506 20549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:48.014716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34691 (local address 127.0.98.1:51435)
0924 05:02:48.014986 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:48.014995 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:02:48.015048 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:02:48.015236 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:02:48.015237 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:02:48.015239 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:02:48.015313 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:48.015314 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:48.015984 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:48.015985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:48.016705 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:48.016707 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:48.017222 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:48.017223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:48.017296 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:48.017299 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 05:02:48.017344 (+ 45us) client_negotiation.cc:240] Negotiation successful
0924 05:02:48.017384 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"mutex_wait_us":62,"thread_start_us":121,"threads_started":1}
I20260924 05:02:48.017495 20539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:48.014858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47459 (local address 127.0.98.3:39713)
0924 05:02:48.014904 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 05:02:48.014906 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:02:48.014914 (+ 8us) server_negotiation.cc:407] Connection header received
0924 05:02:48.014945 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:02:48.014946 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:02:48.014954 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:02:48.015029 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 05:02:48.015236 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:48.015789 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:48.016785 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:48.017147 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:48.017206 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:48.017217 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 05:02:48.017371 (+ 154us) server_negotiation.cc:299] Negotiation successful
0924 05:02:48.017406 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 05:02:48.017606 20266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:48.014716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39713 (local address 127.0.98.1:47459)
0924 05:02:48.014770 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:48.014781 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 05:02:48.014825 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:02:48.015051 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:02:48.015052 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:02:48.015055 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 05:02:48.015130 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:48.015132 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:48.015892 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:48.015893 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:48.016693 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:02:48.016695 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:02:48.017227 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:02:48.017228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:02:48.017301 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:48.017303 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 05:02:48.017350 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 05:02:48.017393 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260924 05:02:48.017992 20349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3dc383b488545d983dff1095f57f894" candidate_uuid: "26540b3808bf4f47805c2120efabc138" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ffb11b801484c168d4ae90c461d4fc4" is_pre_election: true
I20260924 05:02:48.018010 20480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3dc383b488545d983dff1095f57f894" candidate_uuid: "26540b3808bf4f47805c2120efabc138" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc4920b33044157b66dab4f172e634a" is_pre_election: true
I20260924 05:02:48.018124 20349 raft_consensus.cc:2454] T c3dc383b488545d983dff1095f57f894 P 3ffb11b801484c168d4ae90c461d4fc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26540b3808bf4f47805c2120efabc138 in term 0.
I20260924 05:02:48.018134 20480 raft_consensus.cc:2454] T c3dc383b488545d983dff1095f57f894 P 9bc4920b33044157b66dab4f172e634a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26540b3808bf4f47805c2120efabc138 in term 0.
I20260924 05:02:48.018356 20154 leader_election.cc:302] T c3dc383b488545d983dff1095f57f894 P 26540b3808bf4f47805c2120efabc138 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26540b3808bf4f47805c2120efabc138, 3ffb11b801484c168d4ae90c461d4fc4; no voters:
I20260924 05:02:48.018481 20545 raft_consensus.cc:2781] T c3dc383b488545d983dff1095f57f894 P 26540b3808bf4f47805c2120efabc138 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 05:02:48.018533 20545 raft_consensus.cc:482] T c3dc383b488545d983dff1095f57f894 P 26540b3808bf4f47805c2120efabc138 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 05:02:48.018548 20545 raft_consensus.cc:3037] T c3dc383b488545d983dff1095f57f894 P 26540b3808bf4f47805c2120efabc138 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:02:48.019197 20545 raft_consensus.cc:504] T c3dc383b488545d983dff1095f57f894 P 26540b3808bf4f47805c2120efabc138 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ffb11b801484c168d4ae90c461d4fc4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34691 } } peers { permanent_uuid: "9bc4920b33044157b66dab4f172e634a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39713 } } peers { permanent_uuid: "26540b3808bf4f47805c2120efabc138" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46865 } }
I20260924 05:02:48.019307 20545 leader_election.cc:288] T c3dc383b488545d983dff1095f57f894 P 26540b3808bf4f47805c2120efabc138 [CANDIDATE]: Term 1 election: Requested vote from peers 3ffb11b801484c168d4ae90c461d4fc4 (127.0.98.2:34691), 9bc4920b33044157b66dab4f172e634a (127.0.98.3:39713)
I20260924 05:02:48.019471 20480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3dc383b488545d983dff1095f57f894" candidate_uuid: "26540b3808bf4f47805c2120efabc138" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bc4920b33044157b66dab4f172e634a"
I20260924 05:02:48.019495 20349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3dc383b488545d983dff1095f57f894" candidate_uuid: "26540b3808bf4f47805c2120efabc138" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ffb11b801484c168d4ae90c461d4fc4"
I20260924 05:02:48.019543 20480 raft_consensus.cc:3037] T c3dc383b488545d983dff1095f57f894 P 9bc4920b33044157b66dab4f172e634a [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:02:48.019564 20349 raft_consensus.cc:3037] T c3dc383b488545d983dff1095f57f894 P 3ffb11b801484c168d4ae90c461d4fc4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:02:48.020211 20480 raft_consensus.cc:2454] T c3dc383b488545d983dff1095f57f894 P 9bc4920b33044157b66dab4f172e634a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26540b3808bf4f47805c2120efabc138 in term 1.
I20260924 05:02:48.020280 20349 raft_consensus.cc:2454] T c3dc383b488545d983dff1095f57f894 P 3ffb11b801484c168d4ae90c461d4fc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26540b3808bf4f47805c2120efabc138 in term 1.
I20260924 05:02:48.020367 20151 leader_election.cc:302] T c3dc383b488545d983dff1095f57f894 P 26540b3808bf4f47805c2120efabc138 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26540b3808bf4f47805c2120efabc138, 9bc4920b33044157b66dab4f172e634a; no voters:
I20260924 05:02:48.020447 20545 raft_consensus.cc:2781] T c3dc383b488545d983dff1095f57f894 P 26540b3808bf4f47805c2120efabc138 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:02:48.020586 20545 raft_consensus.cc:686] T c3dc383b488545d983dff1095f57f894 P 26540b3808bf4f47805c2120efabc138 [term 1 LEADER]: Becoming Leader. State: Replica: 26540b3808bf4f47805c2120efabc138, State: Running, Role: LEADER
I20260924 05:02:48.020668 20545 consensus_queue.cc:237] T c3dc383b488545d983dff1095f57f894 P 26540b3808bf4f47805c2120efabc138 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ffb11b801484c168d4ae90c461d4fc4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34691 } } peers { permanent_uuid: "9bc4920b33044157b66dab4f172e634a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39713 } } peers { permanent_uuid: "26540b3808bf4f47805c2120efabc138" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46865 } }
I20260924 05:02:48.021502 20076 catalog_manager.cc:5495] T c3dc383b488545d983dff1095f57f894 P 26540b3808bf4f47805c2120efabc138 reported cstate change: term changed from 0 to 1, leader changed from <none> to 26540b3808bf4f47805c2120efabc138 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "26540b3808bf4f47805c2120efabc138" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ffb11b801484c168d4ae90c461d4fc4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9bc4920b33044157b66dab4f172e634a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39713 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26540b3808bf4f47805c2120efabc138" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46865 } health_report { overall_health: HEALTHY } } }
I20260924 05:02:48.032471 20538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:48.030248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34732 (local address 127.0.98.1:46865)
0924 05:02:48.030300 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 05:02:48.030301 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:02:48.030334 (+ 33us) server_negotiation.cc:407] Connection header received
0924 05:02:48.030416 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:02:48.030417 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:02:48.030435 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:02:48.030493 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:02:48.030613 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:48.031080 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:48.031795 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:48.031978 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:48.032031 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:48.032162 (+ 131us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:02:48.032263 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:02:48.032333 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 05:02:48.032351 (+ 18us) server_negotiation.cc:299] Negotiation successful
0924 05:02:48.032373 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 05:02:48.034268 20349 raft_consensus.cc:1261] T c3dc383b488545d983dff1095f57f894 P 3ffb11b801484c168d4ae90c461d4fc4 [term 1 FOLLOWER]: Refusing update from remote peer 26540b3808bf4f47805c2120efabc138: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:02:48.034273 20480 raft_consensus.cc:1261] T c3dc383b488545d983dff1095f57f894 P 9bc4920b33044157b66dab4f172e634a [term 1 FOLLOWER]: Refusing update from remote peer 26540b3808bf4f47805c2120efabc138: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:02:48.034619 20545 consensus_queue.cc:1035] T c3dc383b488545d983dff1095f57f894 P 26540b3808bf4f47805c2120efabc138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bc4920b33044157b66dab4f172e634a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:02:48.034762 20550 consensus_queue.cc:1035] T c3dc383b488545d983dff1095f57f894 P 26540b3808bf4f47805c2120efabc138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ffb11b801484c168d4ae90c461d4fc4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:02:48.037024 20553 mvcc.cc:205] Tried to move back new op lower bound from 7332766384265519104 to 7332766384213217280. Current Snapshot: MvccSnapshot[applied={T|T < 7332766384265519104}]
I20260924 05:02:48.037137 20555 mvcc.cc:205] Tried to move back new op lower bound from 7332766384265519104 to 7332766384213217280. Current Snapshot: MvccSnapshot[applied={T|T < 7332766384265519104}]
I20260924 05:02:48.039499 20074 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44970:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:02:48.039584 20074 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44970:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 05:02:48.040282 20074 catalog_manager.cc:5782] T 00000000000000000000000000000000 P de72094977894f0a809738ff9fb66fd9: Sending DeleteTablet for 3 replicas of tablet c3dc383b488545d983dff1095f57f894
I20260924 05:02:48.040596 20329 tablet_service.cc:1514] Processing DeleteTablet for tablet c3dc383b488545d983dff1095f57f894 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:02:48 UTC) from {username='slave'} at 127.0.0.1:35166
I20260924 05:02:48.040634 20198 tablet_service.cc:1514] Processing DeleteTablet for tablet c3dc383b488545d983dff1095f57f894 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:02:48 UTC) from {username='slave'} at 127.0.0.1:34720
I20260924 05:02:48.040620 20460 tablet_service.cc:1514] Processing DeleteTablet for tablet c3dc383b488545d983dff1095f57f894 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 05:02:48 UTC) from {username='slave'} at 127.0.0.1:35218
I20260924 05:02:48.040894 20564 tablet_replica.cc:330] T c3dc383b488545d983dff1095f57f894 P 9bc4920b33044157b66dab4f172e634a: stopping tablet replica
I20260924 05:02:48.041045 20565 tablet_replica.cc:330] T c3dc383b488545d983dff1095f57f894 P 26540b3808bf4f47805c2120efabc138: stopping tablet replica
I20260924 05:02:48.041044 20564 raft_consensus.cc:2229] T c3dc383b488545d983dff1095f57f894 P 9bc4920b33044157b66dab4f172e634a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:02:48.041127 20565 raft_consensus.cc:2229] T c3dc383b488545d983dff1095f57f894 P 26540b3808bf4f47805c2120efabc138 [term 1 LEADER]: Raft consensus shutting down.
I20260924 05:02:48.041118 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 20136
I20260924 05:02:48.041723 20564 raft_consensus.cc:2258] T c3dc383b488545d983dff1095f57f894 P 9bc4920b33044157b66dab4f172e634a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:02:48.041894 20563 tablet_replica.cc:330] T c3dc383b488545d983dff1095f57f894 P 3ffb11b801484c168d4ae90c461d4fc4: stopping tablet replica
I20260924 05:02:48.041985 20563 raft_consensus.cc:2229] T c3dc383b488545d983dff1095f57f894 P 3ffb11b801484c168d4ae90c461d4fc4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 05:02:48.042078 20563 raft_consensus.cc:2258] T c3dc383b488545d983dff1095f57f894 P 3ffb11b801484c168d4ae90c461d4fc4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 05:02:48.042438 20563 ts_tablet_manager.cc:1905] T c3dc383b488545d983dff1095f57f894 P 3ffb11b801484c168d4ae90c461d4fc4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:02:48.042697 20564 ts_tablet_manager.cc:1905] T c3dc383b488545d983dff1095f57f894 P 9bc4920b33044157b66dab4f172e634a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 05:02:48.044209 20563 ts_tablet_manager.cc:1918] T c3dc383b488545d983dff1095f57f894 P 3ffb11b801484c168d4ae90c461d4fc4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 05:02:48.044275 20564 ts_tablet_manager.cc:1918] T c3dc383b488545d983dff1095f57f894 P 9bc4920b33044157b66dab4f172e634a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 05:02:48.044289 20563 log.cc:1196] T c3dc383b488545d983dff1095f57f894 P 3ffb11b801484c168d4ae90c461d4fc4: Deleting WAL directory at /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225929848825-392-0/minicluster-data/ts-1/wal/wals/c3dc383b488545d983dff1095f57f894
I20260924 05:02:48.044332 20564 log.cc:1196] T c3dc383b488545d983dff1095f57f894 P 9bc4920b33044157b66dab4f172e634a: Deleting WAL directory at /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790225929848825-392-0/minicluster-data/ts-2/wal/wals/c3dc383b488545d983dff1095f57f894
I20260924 05:02:48.044530 20563 ts_tablet_manager.cc:1939] T c3dc383b488545d983dff1095f57f894 P 3ffb11b801484c168d4ae90c461d4fc4: Deleting consensus metadata
I20260924 05:02:48.044531 20564 ts_tablet_manager.cc:1939] T c3dc383b488545d983dff1095f57f894 P 9bc4920b33044157b66dab4f172e634a: Deleting consensus metadata
I20260924 05:02:48.044899 20060 catalog_manager.cc:4841] TS 9bc4920b33044157b66dab4f172e634a (127.0.98.3:39713): tablet c3dc383b488545d983dff1095f57f894 (table test-table [id=97be90e8ecfc4bc988022a59cb51874d]) successfully deleted
I20260924 05:02:48.044937 20063 catalog_manager.cc:4841] TS 3ffb11b801484c168d4ae90c461d4fc4 (127.0.98.2:34691): tablet c3dc383b488545d983dff1095f57f894 (table test-table [id=97be90e8ecfc4bc988022a59cb51874d]) successfully deleted
W20260924 05:02:48.046305 20061 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46865 (error 108)
I20260924 05:02:48.046496 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 20267
I20260924 05:02:48.047274 20534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:48.047146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46865 (local address 127.0.0.1:34746)
0924 05:02:48.047206 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0924 05:02:48.047248 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260924 05:02:48.047402 20061 catalog_manager.cc:4584] TS 26540b3808bf4f47805c2120efabc138 (127.0.98.1:46865): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c3dc383b488545d983dff1095f57f894: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46865: connect: Connection refused (error 111)
I20260924 05:02:48.051020 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 20398
I20260924 05:02:48.055081 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 20045
2026-09-24T05:02:48Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (766 ms)
[----------] 6 tests from EncryptionPolicyTest (7986 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790225929848825-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:02:48 dist-test-slave-70st krb5kdc[20569](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:02:48 dist-test-slave-70st krb5kdc[20569](info): set up 2 sockets
Sep 24 05:02:48 dist-test-slave-70st krb5kdc[20569](info): commencing operation
krb5kdc: starting...
W20260924 05:02:49.005184 392 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790225929848825-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790225929848825-392-0/krb5kdc/test-user.keytab.
Sep 24 05:02:49 dist-test-slave-70st krb5kdc[20569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226169, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:02:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:02:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790225929848825-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790225929848825-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 05:02:49.120862 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790225929848825-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790225929848825-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790225929848825-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36437
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790225929848825-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36437
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:02:49.198998 20582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:02:49.199211 20582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:02:49.199241 20582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:02:49.200387 20582 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:02:49.200429 20582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:02:49.200456 20582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:02:49.200474 20582 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:02:49.200488 20582 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:02:49.201982 20582 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790225929848825-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790225929848825-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36437
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790225929848825-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36437
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20582
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790225929848825-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:02:49.202226 20582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:02:49.202461 20582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:02:49.207264 20590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:02:49.207273 20587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:02:49.207273 20588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:02:49.207602 20582 server_base.cc:1034] running on GCE node
I20260924 05:02:49.207839 20582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:02:49.208115 20582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:02:49.209280 20582 hybrid_clock.cc:648] HybridClock initialized: now 1790226169209257 us; error 34 us; skew 500 ppm
Sep 24 05:02:49 dist-test-slave-70st krb5kdc[20569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226169, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:02:49.211654 20582 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 05:02:49.212042 20582 webserver.cc:460] Webserver started at http://127.0.0.1:46287/ using document root <none> and password file <none>
I20260924 05:02:49.212217 20582 fs_manager.cc:359] Metadata directory not provided
I20260924 05:02:49.212257 20582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:02:49.212345 20582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:02:49.213375 20582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790225929848825-392-0/minicluster-data/master-0/data/instance:
uuid: "ef32c7acb6254aa0bb3c93906f0b2d6a"
format_stamp: "Formatted at 2026-09-24 05:02:49 on dist-test-slave-70st"
server_key: "14f2474dd556901c6364e9d1a1eccf1e"
server_key_iv: "c39ecb20c47df75dd215e0c0f65962b7"
server_key_version: "encryptionkey@0"
I20260924 05:02:49.213762 20582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790225929848825-392-0/minicluster-data/master-0/wal/instance:
uuid: "ef32c7acb6254aa0bb3c93906f0b2d6a"
format_stamp: "Formatted at 2026-09-24 05:02:49 on dist-test-slave-70st"
server_key: "14f2474dd556901c6364e9d1a1eccf1e"
server_key_iv: "c39ecb20c47df75dd215e0c0f65962b7"
server_key_version: "encryptionkey@0"
I20260924 05:02:49.214980 20582 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 05:02:49.215617 20597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:02:49.215758 20582 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:02:49.215839 20582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790225929848825-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790225929848825-392-0/minicluster-data/master-0/wal
uuid: "ef32c7acb6254aa0bb3c93906f0b2d6a"
format_stamp: "Formatted at 2026-09-24 05:02:49 on dist-test-slave-70st"
server_key: "14f2474dd556901c6364e9d1a1eccf1e"
server_key_iv: "c39ecb20c47df75dd215e0c0f65962b7"
server_key_version: "encryptionkey@0"
I20260924 05:02:49.215893 20582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790225929848825-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790225929848825-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790225929848825-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:02:49.235625 20582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:02:49.237372 20582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:02:49.237509 20582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:02:49.241127 20582 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36437
I20260924 05:02:49.241185 20649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36437 every 8 connection(s)
I20260924 05:02:49.241518 20582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:02:49.242357 20650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:02:49.244567 20650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef32c7acb6254aa0bb3c93906f0b2d6a: Bootstrap starting.
I20260924 05:02:49.245100 20650 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef32c7acb6254aa0bb3c93906f0b2d6a: Neither blocks nor log segments found. Creating new log.
I20260924 05:02:49.245349 20650 log.cc:824] T 00000000000000000000000000000000 P ef32c7acb6254aa0bb3c93906f0b2d6a: Log is configured to *not* fsync() on all Append() calls
I20260924 05:02:49.245532 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 20582
I20260924 05:02:49.245652 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790225929848825-392-0/minicluster-data/master-0/wal/instance
I20260924 05:02:49.245834 392 external_mini_cluster.cc:1428] Setting key 3ed86d67ff7cba36494ec3fb8bc6e534
I20260924 05:02:49.246093 20650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef32c7acb6254aa0bb3c93906f0b2d6a: No bootstrap required, opened a new log
I20260924 05:02:49.247787 20650 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef32c7acb6254aa0bb3c93906f0b2d6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef32c7acb6254aa0bb3c93906f0b2d6a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36437 } }
I20260924 05:02:49.247921 20650 raft_consensus.cc:374] T 00000000000000000000000000000000 P ef32c7acb6254aa0bb3c93906f0b2d6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:02:49.247943 20650 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef32c7acb6254aa0bb3c93906f0b2d6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef32c7acb6254aa0bb3c93906f0b2d6a, State: Initialized, Role: FOLLOWER
I20260924 05:02:49.248082 20650 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef32c7acb6254aa0bb3c93906f0b2d6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef32c7acb6254aa0bb3c93906f0b2d6a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36437 } }
I20260924 05:02:49.248167 20650 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef32c7acb6254aa0bb3c93906f0b2d6a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:02:49.248206 20650 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef32c7acb6254aa0bb3c93906f0b2d6a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:02:49.248265 20650 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef32c7acb6254aa0bb3c93906f0b2d6a [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:02:49.249163 20650 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef32c7acb6254aa0bb3c93906f0b2d6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef32c7acb6254aa0bb3c93906f0b2d6a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36437 } }
I20260924 05:02:49.249257 20650 leader_election.cc:302] T 00000000000000000000000000000000 P ef32c7acb6254aa0bb3c93906f0b2d6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef32c7acb6254aa0bb3c93906f0b2d6a; no voters:
I20260924 05:02:49.249411 20650 leader_election.cc:288] T 00000000000000000000000000000000 P ef32c7acb6254aa0bb3c93906f0b2d6a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:02:49.249472 20655 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef32c7acb6254aa0bb3c93906f0b2d6a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:02:49.249583 20655 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef32c7acb6254aa0bb3c93906f0b2d6a [term 1 LEADER]: Becoming Leader. State: Replica: ef32c7acb6254aa0bb3c93906f0b2d6a, State: Running, Role: LEADER
I20260924 05:02:49.249660 20655 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef32c7acb6254aa0bb3c93906f0b2d6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef32c7acb6254aa0bb3c93906f0b2d6a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36437 } }
I20260924 05:02:49.249687 20650 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef32c7acb6254aa0bb3c93906f0b2d6a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:02:49.251353 20656 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef32c7acb6254aa0bb3c93906f0b2d6a [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef32c7acb6254aa0bb3c93906f0b2d6a. Latest consensus state: current_term: 1 leader_uuid: "ef32c7acb6254aa0bb3c93906f0b2d6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef32c7acb6254aa0bb3c93906f0b2d6a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36437 } } }
I20260924 05:02:49.251405 20657 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef32c7acb6254aa0bb3c93906f0b2d6a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ef32c7acb6254aa0bb3c93906f0b2d6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef32c7acb6254aa0bb3c93906f0b2d6a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36437 } } }
I20260924 05:02:49.251623 20656 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef32c7acb6254aa0bb3c93906f0b2d6a [sys.catalog]: This master's current role is: LEADER
I20260924 05:02:49.251641 20657 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef32c7acb6254aa0bb3c93906f0b2d6a [sys.catalog]: This master's current role is: LEADER
W20260924 05:02:49.252096 20669 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ef32c7acb6254aa0bb3c93906f0b2d6a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 05:02:49.252151 20669 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 05:02:49.252377 20671 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:02:49.252553 20671 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:02:49.253950 20671 catalog_manager.cc:1347] Generated new cluster ID: ee83e112c36544498494e275de78f599
Sep 24 05:02:49 dist-test-slave-70st krb5kdc[20569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226169, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 05:02:49.253997 20671 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:02:49.258684 20654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:49.246338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50633 (local address 127.0.0.1:36437)
0924 05:02:49.246507 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 05:02:49.246516 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 05:02:49.246541 (+ 25us) server_negotiation.cc:407] Connection header received
0924 05:02:49.247772 (+ 1231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:02:49.247783 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:02:49.247849 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:02:49.248008 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:02:49.248765 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:49.249981 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:49.250994 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:49.251555 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:49.251659 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:49.254524 (+ 2865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:02:49.254528 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:02:49.254528 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:02:49.254541 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:02:49.256459 (+ 1918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:02:49.256892 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:02:49.256895 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:02:49.256896 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:02:49.257028 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:02:49.257296 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:02:49.257297 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:02:49.257298 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:02:49.257654 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:02:49.257833 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0924 05:02:49.258354 (+ 521us) server_negotiation.cc:299] Negotiation successful
0924 05:02:49.258477 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":71,"threads_started":1}
I20260924 05:02:49.259938 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 05:02:49.262126 20671 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:02:49.262614 20671 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:02:49.267663 20671 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ef32c7acb6254aa0bb3c93906f0b2d6a: Generated new TSK 0
I20260924 05:02:49.267864 20671 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 05:02:49.268641 20654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:49.261448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53269 (local address 127.0.0.1:36437)
0924 05:02:49.261485 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 05:02:49.261487 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:02:49.261649 (+ 162us) server_negotiation.cc:407] Connection header received
0924 05:02:49.261741 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:02:49.261742 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:02:49.261751 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:02:49.261778 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:02:49.262437 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:49.263213 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:49.264247 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:49.264562 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:49.264602 (+ 40us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:49.266079 (+ 1477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:02:49.266081 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:02:49.266081 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:02:49.266087 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:02:49.267503 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:02:49.267882 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:02:49.267883 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:02:49.267883 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:02:49.267945 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:02:49.268170 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:02:49.268171 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:02:49.268171 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:02:49.268302 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:02:49.268382 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 05:02:49.268530 (+ 148us) server_negotiation.cc:299] Negotiation successful
0924 05:02:49.268581 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 05:02:49.270610 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 20582
W20260924 05:02:49.274505 20567 connection.cc:535] client connection to 127.0.0.1:36437 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36437): Connection reset by peer (error 104)
2026-09-24T05:02:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1214 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790225929848825-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:02:49 dist-test-slave-70st krb5kdc[20681](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:02:49 dist-test-slave-70st krb5kdc[20681](info): set up 2 sockets
Sep 24 05:02:49 dist-test-slave-70st krb5kdc[20681](info): commencing operation
krb5kdc: starting...
W20260924 05:02:50.221221 392 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790225929848825-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790225929848825-392-0/krb5kdc/test-user.keytab.
Sep 24 05:02:50 dist-test-slave-70st krb5kdc[20681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226170, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:02:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:02:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790225929848825-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790225929848825-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 05:02:50.329988 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790225929848825-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790225929848825-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790225929848825-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46735
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790225929848825-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46735
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:02:50.407166 20694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:02:50.407359 20694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:02:50.407380 20694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:02:50.408474 20694 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:02:50.408509 20694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:02:50.408520 20694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:02:50.408531 20694 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:02:50.408545 20694 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:02:50.409946 20694 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790225929848825-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790225929848825-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46735
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790225929848825-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46735
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20694
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790225929848825-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:02:50.410128 20694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:02:50.410291 20694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:02:50.414840 20700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:02:50.414812 20702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:02:50.414812 20699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:02:50.414865 20694 server_base.cc:1034] running on GCE node
I20260924 05:02:50.415266 20694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:02:50.415539 20694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:02:50.416674 20694 hybrid_clock.cc:648] HybridClock initialized: now 1790226170416662 us; error 29 us; skew 500 ppm
Sep 24 05:02:50 dist-test-slave-70st krb5kdc[20681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226170, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:02:50.418987 20694 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 05:02:50.419353 20694 webserver.cc:460] Webserver started at http://127.0.0.1:46803/ using document root <none> and password file <none>
I20260924 05:02:50.419524 20694 fs_manager.cc:359] Metadata directory not provided
I20260924 05:02:50.419561 20694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:02:50.419649 20694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:02:50.420601 20694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790225929848825-392-0/minicluster-data/master-0/data/instance:
uuid: "48dd23f139674927a51ca5d585f3ad8f"
format_stamp: "Formatted at 2026-09-24 05:02:50 on dist-test-slave-70st"
server_key: "0c76a4cd7b52cffda93fe1a5283663c1"
server_key_iv: "133e17bc7907a28b6bbdafdbd5cb2bad"
server_key_version: "encryptionkey@0"
I20260924 05:02:50.420876 20694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790225929848825-392-0/minicluster-data/master-0/wal/instance:
uuid: "48dd23f139674927a51ca5d585f3ad8f"
format_stamp: "Formatted at 2026-09-24 05:02:50 on dist-test-slave-70st"
server_key: "0c76a4cd7b52cffda93fe1a5283663c1"
server_key_iv: "133e17bc7907a28b6bbdafdbd5cb2bad"
server_key_version: "encryptionkey@0"
I20260924 05:02:50.422049 20694 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 05:02:50.422637 20709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:02:50.422782 20694 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 05:02:50.422859 20694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790225929848825-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790225929848825-392-0/minicluster-data/master-0/wal
uuid: "48dd23f139674927a51ca5d585f3ad8f"
format_stamp: "Formatted at 2026-09-24 05:02:50 on dist-test-slave-70st"
server_key: "0c76a4cd7b52cffda93fe1a5283663c1"
server_key_iv: "133e17bc7907a28b6bbdafdbd5cb2bad"
server_key_version: "encryptionkey@0"
I20260924 05:02:50.422915 20694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790225929848825-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790225929848825-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790225929848825-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:02:50.436430 20694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:02:50.438128 20694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:02:50.438230 20694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:02:50.441809 20694 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46735
I20260924 05:02:50.441843 20761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46735 every 8 connection(s)
I20260924 05:02:50.442157 20694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:02:50.443004 20762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:02:50.443928 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 20694
I20260924 05:02:50.444018 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790225929848825-392-0/minicluster-data/master-0/wal/instance
I20260924 05:02:50.444178 392 external_mini_cluster.cc:1428] Setting key 265c8ee75178e5d78315cb8f021c49eb
I20260924 05:02:50.445660 20762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48dd23f139674927a51ca5d585f3ad8f: Bootstrap starting.
I20260924 05:02:50.446215 20762 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 48dd23f139674927a51ca5d585f3ad8f: Neither blocks nor log segments found. Creating new log.
I20260924 05:02:50.446477 20762 log.cc:824] T 00000000000000000000000000000000 P 48dd23f139674927a51ca5d585f3ad8f: Log is configured to *not* fsync() on all Append() calls
I20260924 05:02:50.447132 20762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48dd23f139674927a51ca5d585f3ad8f: No bootstrap required, opened a new log
I20260924 05:02:50.448815 20762 raft_consensus.cc:348] T 00000000000000000000000000000000 P 48dd23f139674927a51ca5d585f3ad8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48dd23f139674927a51ca5d585f3ad8f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46735 } }
I20260924 05:02:50.448945 20762 raft_consensus.cc:374] T 00000000000000000000000000000000 P 48dd23f139674927a51ca5d585f3ad8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:02:50.448976 20762 raft_consensus.cc:729] T 00000000000000000000000000000000 P 48dd23f139674927a51ca5d585f3ad8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48dd23f139674927a51ca5d585f3ad8f, State: Initialized, Role: FOLLOWER
I20260924 05:02:50.449103 20762 consensus_queue.cc:260] T 00000000000000000000000000000000 P 48dd23f139674927a51ca5d585f3ad8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48dd23f139674927a51ca5d585f3ad8f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46735 } }
I20260924 05:02:50.449178 20762 raft_consensus.cc:388] T 00000000000000000000000000000000 P 48dd23f139674927a51ca5d585f3ad8f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:02:50.449216 20762 raft_consensus.cc:482] T 00000000000000000000000000000000 P 48dd23f139674927a51ca5d585f3ad8f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:02:50.449258 20762 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 48dd23f139674927a51ca5d585f3ad8f [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:02:50.449260 20765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:50.444668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52979 (local address 127.0.0.1:46735)
0924 05:02:50.444847 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 05:02:50.444856 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 05:02:50.444887 (+ 31us) server_negotiation.cc:407] Connection header received
0924 05:02:50.445372 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:02:50.445381 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:02:50.445465 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:02:50.445609 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:02:50.445788 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:50.446747 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:50.447536 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:50.447953 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:50.448047 (+ 94us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:50.448674 (+ 627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:02:50.448677 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:02:50.448677 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:02:50.448824 (+ 147us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:02:50.448853 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:02:50.448864 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:02:50.448940 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 05:02:50.449038 (+ 98us) server_negotiation.cc:299] Negotiation successful
0924 05:02:50.449067 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":74,"threads_started":1}
I20260924 05:02:50.449807 20762 raft_consensus.cc:504] T 00000000000000000000000000000000 P 48dd23f139674927a51ca5d585f3ad8f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48dd23f139674927a51ca5d585f3ad8f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46735 } }
I20260924 05:02:50.449909 20762 leader_election.cc:302] T 00000000000000000000000000000000 P 48dd23f139674927a51ca5d585f3ad8f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 48dd23f139674927a51ca5d585f3ad8f; no voters:
I20260924 05:02:50.450067 20762 leader_election.cc:288] T 00000000000000000000000000000000 P 48dd23f139674927a51ca5d585f3ad8f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:02:50.450102 20767 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 48dd23f139674927a51ca5d585f3ad8f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:02:50.450207 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 05:02:50.450299 20762 sys_catalog.cc:564] T 00000000000000000000000000000000 P 48dd23f139674927a51ca5d585f3ad8f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:02:50.451359 20767 raft_consensus.cc:686] T 00000000000000000000000000000000 P 48dd23f139674927a51ca5d585f3ad8f [term 1 LEADER]: Becoming Leader. State: Replica: 48dd23f139674927a51ca5d585f3ad8f, State: Running, Role: LEADER
I20260924 05:02:50.451459 20767 consensus_queue.cc:237] T 00000000000000000000000000000000 P 48dd23f139674927a51ca5d585f3ad8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48dd23f139674927a51ca5d585f3ad8f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46735 } }
I20260924 05:02:50.451933 20768 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48dd23f139674927a51ca5d585f3ad8f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "48dd23f139674927a51ca5d585f3ad8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48dd23f139674927a51ca5d585f3ad8f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46735 } } }
I20260924 05:02:50.451951 20783 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48dd23f139674927a51ca5d585f3ad8f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 48dd23f139674927a51ca5d585f3ad8f. Latest consensus state: current_term: 1 leader_uuid: "48dd23f139674927a51ca5d585f3ad8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48dd23f139674927a51ca5d585f3ad8f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46735 } } }
I20260924 05:02:50.452726 20768 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48dd23f139674927a51ca5d585f3ad8f [sys.catalog]: This master's current role is: LEADER
I20260924 05:02:50.452734 20783 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48dd23f139674927a51ca5d585f3ad8f [sys.catalog]: This master's current role is: LEADER
I20260924 05:02:50.452934 20788 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:02:50.453387 20788 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:02:50.454782 20765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:50.451463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51341 (local address 127.0.0.1:46735)
0924 05:02:50.451496 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 05:02:50.451498 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:02:50.451598 (+ 100us) server_negotiation.cc:407] Connection header received
0924 05:02:50.451655 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:02:50.451656 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:02:50.451666 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:02:50.451706 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:02:50.451868 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:50.452421 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:50.453464 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:50.453737 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:50.453804 (+ 67us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:50.454573 (+ 769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:02:50.454574 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:02:50.454574 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:02:50.454578 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:02:50.454592 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:02:50.454596 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:02:50.454644 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 05:02:50.454710 (+ 66us) server_negotiation.cc:299] Negotiation successful
0924 05:02:50.454719 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260924 05:02:50.455035 20788 catalog_manager.cc:1347] Generated new cluster ID: ce09e7db1ba746d6aa080c4d1910fc26
I20260924 05:02:50.455087 20788 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:02:50.474839 20788 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:02:50.475569 20788 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:02:50.485591 20788 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 48dd23f139674927a51ca5d585f3ad8f: Generated new TSK 0
I20260924 05:02:50.485808 20788 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 05:02:50.493687 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 20694
W20260924 05:02:50.497329 20679 connection.cc:535] client connection to 127.0.0.1:46735 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46735): Connection reset by peer (error 104)
2026-09-24T05:02:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1222 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790225929848825-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:02:50 dist-test-slave-70st krb5kdc[20793](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:02:50 dist-test-slave-70st krb5kdc[20793](info): set up 2 sockets
Sep 24 05:02:50 dist-test-slave-70st krb5kdc[20793](info): commencing operation
krb5kdc: starting...
W20260924 05:02:51.441099 392 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790225929848825-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790225929848825-392-0/krb5kdc/test-user.keytab.
Sep 24 05:02:51 dist-test-slave-70st krb5kdc[20793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226171, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:02:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:02:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790225929848825-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790225929848825-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 05:02:51.550633 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790225929848825-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790225929848825-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790225929848825-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34557
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790225929848825-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34557
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:02:51.629174 20806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:02:51.629379 20806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:02:51.629411 20806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:02:51.630561 20806 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:02:51.630602 20806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:02:51.630621 20806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:02:51.630636 20806 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:02:51.630652 20806 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:02:51.632097 20806 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790225929848825-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790225929848825-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34557
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790225929848825-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20806
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790225929848825-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:02:51.632324 20806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:02:51.632534 20806 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:02:51.636976 20812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:02:51.636989 20811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:02:51.637053 20814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:02:51.637315 20806 server_base.cc:1034] running on GCE node
I20260924 05:02:51.637589 20806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:02:51.637837 20806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:02:51.638998 20806 hybrid_clock.cc:648] HybridClock initialized: now 1790226171638984 us; error 29 us; skew 500 ppm
Sep 24 05:02:51 dist-test-slave-70st krb5kdc[20793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226171, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:02:51.641321 20806 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 05:02:51.641791 20806 webserver.cc:460] Webserver started at http://127.0.0.1:41491/ using document root <none> and password file <none>
I20260924 05:02:51.642007 20806 fs_manager.cc:359] Metadata directory not provided
I20260924 05:02:51.642055 20806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:02:51.642161 20806 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:02:51.643210 20806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790225929848825-392-0/minicluster-data/master-0/data/instance:
uuid: "3b197b39a60c4b30b175ff3e6836971a"
format_stamp: "Formatted at 2026-09-24 05:02:51 on dist-test-slave-70st"
server_key: "430325d6c4a5fc03b43b7cd3a744e711"
server_key_iv: "4e5ae429c1a239fb0864e9acbfab5b43"
server_key_version: "encryptionkey@0"
I20260924 05:02:51.643550 20806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790225929848825-392-0/minicluster-data/master-0/wal/instance:
uuid: "3b197b39a60c4b30b175ff3e6836971a"
format_stamp: "Formatted at 2026-09-24 05:02:51 on dist-test-slave-70st"
server_key: "430325d6c4a5fc03b43b7cd3a744e711"
server_key_iv: "4e5ae429c1a239fb0864e9acbfab5b43"
server_key_version: "encryptionkey@0"
I20260924 05:02:51.644906 20806 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 05:02:51.645604 20821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:02:51.645735 20806 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:02:51.645854 20806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790225929848825-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790225929848825-392-0/minicluster-data/master-0/wal
uuid: "3b197b39a60c4b30b175ff3e6836971a"
format_stamp: "Formatted at 2026-09-24 05:02:51 on dist-test-slave-70st"
server_key: "430325d6c4a5fc03b43b7cd3a744e711"
server_key_iv: "4e5ae429c1a239fb0864e9acbfab5b43"
server_key_version: "encryptionkey@0"
I20260924 05:02:51.645918 20806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790225929848825-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790225929848825-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790225929848825-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:02:51.649034 20806 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:02:51.650684 20806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:02:51.650794 20806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:02:51.654404 20806 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34557
I20260924 05:02:51.654439 20873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34557 every 8 connection(s)
I20260924 05:02:51.654737 20806 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:02:51.655619 20874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:02:51.657735 20874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b197b39a60c4b30b175ff3e6836971a: Bootstrap starting.
I20260924 05:02:51.658218 20874 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b197b39a60c4b30b175ff3e6836971a: Neither blocks nor log segments found. Creating new log.
I20260924 05:02:51.658442 20874 log.cc:824] T 00000000000000000000000000000000 P 3b197b39a60c4b30b175ff3e6836971a: Log is configured to *not* fsync() on all Append() calls
I20260924 05:02:51.659055 20874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b197b39a60c4b30b175ff3e6836971a: No bootstrap required, opened a new log
I20260924 05:02:51.660564 20874 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b197b39a60c4b30b175ff3e6836971a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b197b39a60c4b30b175ff3e6836971a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34557 } }
I20260924 05:02:51.660696 20874 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3b197b39a60c4b30b175ff3e6836971a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:02:51.660717 20874 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b197b39a60c4b30b175ff3e6836971a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b197b39a60c4b30b175ff3e6836971a, State: Initialized, Role: FOLLOWER
I20260924 05:02:51.660815 20874 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b197b39a60c4b30b175ff3e6836971a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b197b39a60c4b30b175ff3e6836971a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34557 } }
I20260924 05:02:51.660864 20874 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3b197b39a60c4b30b175ff3e6836971a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:02:51.660890 20874 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b197b39a60c4b30b175ff3e6836971a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:02:51.660918 20874 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b197b39a60c4b30b175ff3e6836971a [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:02:51.661381 20874 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b197b39a60c4b30b175ff3e6836971a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b197b39a60c4b30b175ff3e6836971a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34557 } }
I20260924 05:02:51.661501 20874 leader_election.cc:302] T 00000000000000000000000000000000 P 3b197b39a60c4b30b175ff3e6836971a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3b197b39a60c4b30b175ff3e6836971a; no voters:
I20260924 05:02:51.661628 20874 leader_election.cc:288] T 00000000000000000000000000000000 P 3b197b39a60c4b30b175ff3e6836971a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:02:51.661674 20877 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b197b39a60c4b30b175ff3e6836971a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:02:51.661846 20877 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b197b39a60c4b30b175ff3e6836971a [term 1 LEADER]: Becoming Leader. State: Replica: 3b197b39a60c4b30b175ff3e6836971a, State: Running, Role: LEADER
I20260924 05:02:51.661844 20874 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b197b39a60c4b30b175ff3e6836971a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:02:51.661988 20877 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b197b39a60c4b30b175ff3e6836971a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b197b39a60c4b30b175ff3e6836971a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34557 } }
I20260924 05:02:51.662309 20878 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b197b39a60c4b30b175ff3e6836971a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3b197b39a60c4b30b175ff3e6836971a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b197b39a60c4b30b175ff3e6836971a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34557 } } }
I20260924 05:02:51.662307 20879 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b197b39a60c4b30b175ff3e6836971a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b197b39a60c4b30b175ff3e6836971a. Latest consensus state: current_term: 1 leader_uuid: "3b197b39a60c4b30b175ff3e6836971a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b197b39a60c4b30b175ff3e6836971a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34557 } } }
I20260924 05:02:51.662400 20878 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b197b39a60c4b30b175ff3e6836971a [sys.catalog]: This master's current role is: LEADER
I20260924 05:02:51.662408 20879 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b197b39a60c4b30b175ff3e6836971a [sys.catalog]: This master's current role is: LEADER
I20260924 05:02:51.662581 20881 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:02:51.663065 20881 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:02:51.664558 20881 catalog_manager.cc:1347] Generated new cluster ID: 74fe1c66afbe4254ac46af5d2ff7935a
I20260924 05:02:51.664609 20881 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:02:51.664762 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 20806
I20260924 05:02:51.664896 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790225929848825-392-0/minicluster-data/master-0/wal/instance
I20260924 05:02:51.665042 392 external_mini_cluster.cc:1428] Setting key 69290ffcee8fd6299e1156f98d6ecd3b
I20260924 05:02:51.671159 20897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:51.665473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52121 (local address 127.0.0.1:34557)
0924 05:02:51.669542 (+ 4069us) server_negotiation.cc:206] Beginning negotiation
0924 05:02:51.669551 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 05:02:51.669765 (+ 214us) server_negotiation.cc:407] Connection header received
0924 05:02:51.669895 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:02:51.669902 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:02:51.669945 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:02:51.670097 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:02:51.670723 (+ 626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:02:51.670727 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:02:51.670727 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:02:51.670732 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:02:51.670775 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:02:51.670797 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:02:51.670855 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 05:02:51.670925 (+ 70us) server_negotiation.cc:299] Negotiation successful
0924 05:02:51.670964 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4020,"thread_start_us":63,"threads_started":1}
I20260924 05:02:51.672019 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 05:02:51.674391 20897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:51.673445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34065 (local address 127.0.0.1:34557)
0924 05:02:51.673477 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0924 05:02:51.673478 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:02:51.673581 (+ 103us) server_negotiation.cc:407] Connection header received
0924 05:02:51.673628 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:02:51.673629 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:02:51.673633 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:02:51.673684 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:02:51.674213 (+ 529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:02:51.674214 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:02:51.674215 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:02:51.674218 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:02:51.674232 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:02:51.674236 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:02:51.674281 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 05:02:51.674323 (+ 42us) server_negotiation.cc:299] Negotiation successful
0924 05:02:51.674334 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260924 05:02:51.688931 20881 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:02:51.689118 20881 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:02:51.696729 20881 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3b197b39a60c4b30b175ff3e6836971a: Generated new TSK 0
I20260924 05:02:51.696918 20881 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 05:02:51.701210 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 20806
2026-09-24T05:02:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1207 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790225929848825-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:02:51 dist-test-slave-70st krb5kdc[20905](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:02:51 dist-test-slave-70st krb5kdc[20905](info): set up 2 sockets
Sep 24 05:02:51 dist-test-slave-70st krb5kdc[20905](info): commencing operation
krb5kdc: starting...
W20260924 05:02:52.653307 392 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790225929848825-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790225929848825-392-0/krb5kdc/test-user.keytab.
Sep 24 05:02:52 dist-test-slave-70st krb5kdc[20905](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226172, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:02:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:02:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790225929848825-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790225929848825-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 05:02:52.764452 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790225929848825-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790225929848825-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790225929848825-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39939
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790225929848825-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39939
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:02:52.842996 20918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:02:52.843204 20918 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:02:52.843232 20918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:02:52.844342 20918 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:02:52.844383 20918 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:02:52.844400 20918 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:02:52.844416 20918 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:02:52.844430 20918 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:02:52.845867 20918 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790225929848825-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790225929848825-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39939
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790225929848825-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39939
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20918
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790225929848825-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:02:52.846098 20918 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:02:52.846299 20918 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:02:52.850834 20923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:02:52.850893 20918 server_base.cc:1034] running on GCE node
W20260924 05:02:52.850864 20924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:02:52.850826 20926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:02:52.851284 20918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:02:52.851544 20918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:02:52.852689 20918 hybrid_clock.cc:648] HybridClock initialized: now 1790226172852676 us; error 30 us; skew 500 ppm
Sep 24 05:02:52 dist-test-slave-70st krb5kdc[20905](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226172, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:02:52.855098 20918 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 05:02:52.855502 20918 webserver.cc:460] Webserver started at http://127.0.0.1:39155/ using document root <none> and password file <none>
I20260924 05:02:52.855683 20918 fs_manager.cc:359] Metadata directory not provided
I20260924 05:02:52.855724 20918 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:02:52.855844 20918 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:02:52.856827 20918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790225929848825-392-0/minicluster-data/master-0/data/instance:
uuid: "9ed6b2a17bbe46d8b4eb8d613e4d8768"
format_stamp: "Formatted at 2026-09-24 05:02:52 on dist-test-slave-70st"
server_key: "38e9e4518189d61b1b480b49b649f667"
server_key_iv: "fe93fbf777bf05f67e7c169bc4810fd8"
server_key_version: "encryptionkey@0"
I20260924 05:02:52.857137 20918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790225929848825-392-0/minicluster-data/master-0/wal/instance:
uuid: "9ed6b2a17bbe46d8b4eb8d613e4d8768"
format_stamp: "Formatted at 2026-09-24 05:02:52 on dist-test-slave-70st"
server_key: "38e9e4518189d61b1b480b49b649f667"
server_key_iv: "fe93fbf777bf05f67e7c169bc4810fd8"
server_key_version: "encryptionkey@0"
I20260924 05:02:52.858350 20918 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 05:02:52.859025 20933 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:02:52.859169 20918 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 05:02:52.859243 20918 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790225929848825-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790225929848825-392-0/minicluster-data/master-0/wal
uuid: "9ed6b2a17bbe46d8b4eb8d613e4d8768"
format_stamp: "Formatted at 2026-09-24 05:02:52 on dist-test-slave-70st"
server_key: "38e9e4518189d61b1b480b49b649f667"
server_key_iv: "fe93fbf777bf05f67e7c169bc4810fd8"
server_key_version: "encryptionkey@0"
I20260924 05:02:52.859311 20918 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790225929848825-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790225929848825-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790225929848825-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:02:52.867012 20918 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:02:52.868973 20918 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:02:52.869088 20918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:02:52.872524 20918 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39939
I20260924 05:02:52.872573 20985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39939 every 8 connection(s)
I20260924 05:02:52.872859 20918 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:02:52.873672 20986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:02:52.875803 20986 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ed6b2a17bbe46d8b4eb8d613e4d8768: Bootstrap starting.
I20260924 05:02:52.876320 20986 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9ed6b2a17bbe46d8b4eb8d613e4d8768: Neither blocks nor log segments found. Creating new log.
I20260924 05:02:52.876564 20986 log.cc:824] T 00000000000000000000000000000000 P 9ed6b2a17bbe46d8b4eb8d613e4d8768: Log is configured to *not* fsync() on all Append() calls
I20260924 05:02:52.877298 20986 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ed6b2a17bbe46d8b4eb8d613e4d8768: No bootstrap required, opened a new log
I20260924 05:02:52.878818 20986 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ed6b2a17bbe46d8b4eb8d613e4d8768 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ed6b2a17bbe46d8b4eb8d613e4d8768" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39939 } }
I20260924 05:02:52.878953 20986 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9ed6b2a17bbe46d8b4eb8d613e4d8768 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:02:52.878966 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 20918
I20260924 05:02:52.878985 20986 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ed6b2a17bbe46d8b4eb8d613e4d8768 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ed6b2a17bbe46d8b4eb8d613e4d8768, State: Initialized, Role: FOLLOWER
I20260924 05:02:52.879063 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790225929848825-392-0/minicluster-data/master-0/wal/instance
I20260924 05:02:52.879129 20986 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ed6b2a17bbe46d8b4eb8d613e4d8768 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ed6b2a17bbe46d8b4eb8d613e4d8768" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39939 } }
I20260924 05:02:52.879207 20986 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9ed6b2a17bbe46d8b4eb8d613e4d8768 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:02:52.879228 392 external_mini_cluster.cc:1428] Setting key 12c3ce7baba3fc31316221639c63dc4d
I20260924 05:02:52.879240 20986 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9ed6b2a17bbe46d8b4eb8d613e4d8768 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:02:52.879280 20986 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ed6b2a17bbe46d8b4eb8d613e4d8768 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:02:52.880254 20986 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9ed6b2a17bbe46d8b4eb8d613e4d8768 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ed6b2a17bbe46d8b4eb8d613e4d8768" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39939 } }
I20260924 05:02:52.880359 20986 leader_election.cc:302] T 00000000000000000000000000000000 P 9ed6b2a17bbe46d8b4eb8d613e4d8768 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9ed6b2a17bbe46d8b4eb8d613e4d8768; no voters:
I20260924 05:02:52.880517 20986 leader_election.cc:288] T 00000000000000000000000000000000 P 9ed6b2a17bbe46d8b4eb8d613e4d8768 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:02:52.880542 20991 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9ed6b2a17bbe46d8b4eb8d613e4d8768 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:02:52.880636 20991 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9ed6b2a17bbe46d8b4eb8d613e4d8768 [term 1 LEADER]: Becoming Leader. State: Replica: 9ed6b2a17bbe46d8b4eb8d613e4d8768, State: Running, Role: LEADER
I20260924 05:02:52.880757 20986 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ed6b2a17bbe46d8b4eb8d613e4d8768 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:02:52.880764 20991 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ed6b2a17bbe46d8b4eb8d613e4d8768 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ed6b2a17bbe46d8b4eb8d613e4d8768" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39939 } }
I20260924 05:02:52.881146 20991 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ed6b2a17bbe46d8b4eb8d613e4d8768 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ed6b2a17bbe46d8b4eb8d613e4d8768. Latest consensus state: current_term: 1 leader_uuid: "9ed6b2a17bbe46d8b4eb8d613e4d8768" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ed6b2a17bbe46d8b4eb8d613e4d8768" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39939 } } }
I20260924 05:02:52.881273 20991 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ed6b2a17bbe46d8b4eb8d613e4d8768 [sys.catalog]: This master's current role is: LEADER
I20260924 05:02:52.881654 21000 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:02:52.882207 21000 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:02:52.882998 20992 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ed6b2a17bbe46d8b4eb8d613e4d8768 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9ed6b2a17bbe46d8b4eb8d613e4d8768" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ed6b2a17bbe46d8b4eb8d613e4d8768" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39939 } } }
I20260924 05:02:52.883136 20992 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ed6b2a17bbe46d8b4eb8d613e4d8768 [sys.catalog]: This master's current role is: LEADER
I20260924 05:02:52.883642 21000 catalog_manager.cc:1347] Generated new cluster ID: 2ed2bcb411244209b5f0d0d50c0bd8dc
I20260924 05:02:52.883698 21000 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 05:02:52 dist-test-slave-70st krb5kdc[20905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226172, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 05:02:52.890530 20990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:52.879672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54817 (local address 127.0.0.1:39939)
0924 05:02:52.879908 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0924 05:02:52.879918 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 05:02:52.879967 (+ 49us) server_negotiation.cc:407] Connection header received
0924 05:02:52.880083 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:02:52.880090 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:02:52.880152 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:02:52.880302 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:02:52.881095 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:52.882624 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:52.883575 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:52.884150 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:52.884249 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:52.886948 (+ 2699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:02:52.886952 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:02:52.886953 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:02:52.886964 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:02:52.888907 (+ 1943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:02:52.889263 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:02:52.889266 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:02:52.889266 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:02:52.889385 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:02:52.889617 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:02:52.889617 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:02:52.889618 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:02:52.889940 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:02:52.890104 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0924 05:02:52.890172 (+ 68us) server_negotiation.cc:299] Negotiation successful
0924 05:02:52.890310 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":127,"threads_started":1}
I20260924 05:02:52.891417 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 05:02:52.899502 20990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:52.892881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51245 (local address 127.0.0.1:39939)
0924 05:02:52.892920 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 05:02:52.892922 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:02:52.893043 (+ 121us) server_negotiation.cc:407] Connection header received
0924 05:02:52.893099 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:02:52.893100 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:02:52.893110 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:02:52.893153 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:02:52.893744 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:52.894442 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:52.895446 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:52.895724 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:52.895773 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:52.897036 (+ 1263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:02:52.897037 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:02:52.897037 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:02:52.897043 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:02:52.898503 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:02:52.898864 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:02:52.898865 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:02:52.898865 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:02:52.898911 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:02:52.899077 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:02:52.899077 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:02:52.899077 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:02:52.899180 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:02:52.899269 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 05:02:52.899357 (+ 88us) server_negotiation.cc:299] Negotiation successful
0924 05:02:52.899419 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 05:02:52.910166 21000 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:02:52.910908 21000 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:02:52.924005 21000 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9ed6b2a17bbe46d8b4eb8d613e4d8768: Generated new TSK 0
I20260924 05:02:52.924240 21000 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 05:02:52.932039 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 20918
2026-09-24T05:02:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1230 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790225929848825-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:02:52 dist-test-slave-70st krb5kdc[21017](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:02:52 dist-test-slave-70st krb5kdc[21017](info): set up 2 sockets
Sep 24 05:02:52 dist-test-slave-70st krb5kdc[21017](info): commencing operation
krb5kdc: starting...
W20260924 05:02:53.885145 392 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790225929848825-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790225929848825-392-0/krb5kdc/test-user.keytab.
Sep 24 05:02:53 dist-test-slave-70st krb5kdc[21017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226173, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:02:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:02:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790225929848825-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790225929848825-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 05:02:53.995668 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790225929848825-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790225929848825-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790225929848825-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37969
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790225929848825-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37969
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:02:54.074455 21030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:02:54.074669 21030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:02:54.074700 21030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:02:54.075875 21030 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:02:54.075922 21030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:02:54.075939 21030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:02:54.075956 21030 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:02:54.075970 21030 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:02:54.077463 21030 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790225929848825-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790225929848825-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37969
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790225929848825-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21030
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790225929848825-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:02:54.077713 21030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:02:54.077919 21030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:02:54.082993 21038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:02:54.083050 21030 server_base.cc:1034] running on GCE node
W20260924 05:02:54.083007 21036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:02:54.083009 21035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:02:54.083369 21030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:02:54.083658 21030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:02:54.084798 21030 hybrid_clock.cc:648] HybridClock initialized: now 1790226174084784 us; error 30 us; skew 500 ppm
Sep 24 05:02:54 dist-test-slave-70st krb5kdc[21017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226174, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:02:54.087203 21030 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 05:02:54.087634 21030 webserver.cc:460] Webserver started at http://127.0.0.1:34253/ using document root <none> and password file <none>
I20260924 05:02:54.087836 21030 fs_manager.cc:359] Metadata directory not provided
I20260924 05:02:54.087883 21030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:02:54.088002 21030 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:02:54.089010 21030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790225929848825-392-0/minicluster-data/master-0/data/instance:
uuid: "c6f794a9b8c74d74a1bea28b8adb7d0b"
format_stamp: "Formatted at 2026-09-24 05:02:54 on dist-test-slave-70st"
server_key: "12ae1cb85a404a3f0c5da4bb5ece0617"
server_key_iv: "566a4a9f2f213e5e9ce9adb052726845"
server_key_version: "encryptionkey@0"
I20260924 05:02:54.089327 21030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790225929848825-392-0/minicluster-data/master-0/wal/instance:
uuid: "c6f794a9b8c74d74a1bea28b8adb7d0b"
format_stamp: "Formatted at 2026-09-24 05:02:54 on dist-test-slave-70st"
server_key: "12ae1cb85a404a3f0c5da4bb5ece0617"
server_key_iv: "566a4a9f2f213e5e9ce9adb052726845"
server_key_version: "encryptionkey@0"
I20260924 05:02:54.090549 21030 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 05:02:54.091173 21045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:02:54.091320 21030 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:02:54.091396 21030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790225929848825-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790225929848825-392-0/minicluster-data/master-0/wal
uuid: "c6f794a9b8c74d74a1bea28b8adb7d0b"
format_stamp: "Formatted at 2026-09-24 05:02:54 on dist-test-slave-70st"
server_key: "12ae1cb85a404a3f0c5da4bb5ece0617"
server_key_iv: "566a4a9f2f213e5e9ce9adb052726845"
server_key_version: "encryptionkey@0"
I20260924 05:02:54.091454 21030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790225929848825-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790225929848825-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790225929848825-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:02:54.099310 21030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:02:54.100898 21030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:02:54.101015 21030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:02:54.104563 21030 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37969
I20260924 05:02:54.104591 21097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37969 every 8 connection(s)
I20260924 05:02:54.104900 21030 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:02:54.105684 21098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:02:54.107832 21098 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6f794a9b8c74d74a1bea28b8adb7d0b: Bootstrap starting.
I20260924 05:02:54.108373 21098 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c6f794a9b8c74d74a1bea28b8adb7d0b: Neither blocks nor log segments found. Creating new log.
I20260924 05:02:54.108616 21098 log.cc:824] T 00000000000000000000000000000000 P c6f794a9b8c74d74a1bea28b8adb7d0b: Log is configured to *not* fsync() on all Append() calls
I20260924 05:02:54.109288 21098 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6f794a9b8c74d74a1bea28b8adb7d0b: No bootstrap required, opened a new log
I20260924 05:02:54.109779 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 21030
I20260924 05:02:54.109882 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790225929848825-392-0/minicluster-data/master-0/wal/instance
I20260924 05:02:54.110041 392 external_mini_cluster.cc:1428] Setting key 38843692706a601526778e9174e42c3d
I20260924 05:02:54.111323 21098 raft_consensus.cc:348] T 00000000000000000000000000000000 P c6f794a9b8c74d74a1bea28b8adb7d0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6f794a9b8c74d74a1bea28b8adb7d0b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37969 } }
I20260924 05:02:54.111509 21098 raft_consensus.cc:374] T 00000000000000000000000000000000 P c6f794a9b8c74d74a1bea28b8adb7d0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:02:54.111542 21098 raft_consensus.cc:729] T 00000000000000000000000000000000 P c6f794a9b8c74d74a1bea28b8adb7d0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6f794a9b8c74d74a1bea28b8adb7d0b, State: Initialized, Role: FOLLOWER
I20260924 05:02:54.111663 21098 consensus_queue.cc:260] T 00000000000000000000000000000000 P c6f794a9b8c74d74a1bea28b8adb7d0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6f794a9b8c74d74a1bea28b8adb7d0b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37969 } }
I20260924 05:02:54.111740 21098 raft_consensus.cc:388] T 00000000000000000000000000000000 P c6f794a9b8c74d74a1bea28b8adb7d0b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:02:54.111771 21098 raft_consensus.cc:482] T 00000000000000000000000000000000 P c6f794a9b8c74d74a1bea28b8adb7d0b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:02:54.111853 21098 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c6f794a9b8c74d74a1bea28b8adb7d0b [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:02:54.112514 21098 raft_consensus.cc:504] T 00000000000000000000000000000000 P c6f794a9b8c74d74a1bea28b8adb7d0b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6f794a9b8c74d74a1bea28b8adb7d0b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37969 } }
I20260924 05:02:54.112617 21098 leader_election.cc:302] T 00000000000000000000000000000000 P c6f794a9b8c74d74a1bea28b8adb7d0b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c6f794a9b8c74d74a1bea28b8adb7d0b; no voters:
I20260924 05:02:54.112771 21098 leader_election.cc:288] T 00000000000000000000000000000000 P c6f794a9b8c74d74a1bea28b8adb7d0b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:02:54.112821 21103 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c6f794a9b8c74d74a1bea28b8adb7d0b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:02:54.113006 21098 sys_catalog.cc:564] T 00000000000000000000000000000000 P c6f794a9b8c74d74a1bea28b8adb7d0b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:02:54.113014 21103 raft_consensus.cc:686] T 00000000000000000000000000000000 P c6f794a9b8c74d74a1bea28b8adb7d0b [term 1 LEADER]: Becoming Leader. State: Replica: c6f794a9b8c74d74a1bea28b8adb7d0b, State: Running, Role: LEADER
I20260924 05:02:54.113184 21103 consensus_queue.cc:237] T 00000000000000000000000000000000 P c6f794a9b8c74d74a1bea28b8adb7d0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6f794a9b8c74d74a1bea28b8adb7d0b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37969 } }
I20260924 05:02:54.113551 21104 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6f794a9b8c74d74a1bea28b8adb7d0b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c6f794a9b8c74d74a1bea28b8adb7d0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6f794a9b8c74d74a1bea28b8adb7d0b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37969 } } }
I20260924 05:02:54.113617 21104 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6f794a9b8c74d74a1bea28b8adb7d0b [sys.catalog]: This master's current role is: LEADER
I20260924 05:02:54.113777 21105 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6f794a9b8c74d74a1bea28b8adb7d0b [sys.catalog]: SysCatalogTable state changed. Reason: New leader c6f794a9b8c74d74a1bea28b8adb7d0b. Latest consensus state: current_term: 1 leader_uuid: "c6f794a9b8c74d74a1bea28b8adb7d0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6f794a9b8c74d74a1bea28b8adb7d0b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37969 } } }
I20260924 05:02:54.113826 21105 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6f794a9b8c74d74a1bea28b8adb7d0b [sys.catalog]: This master's current role is: LEADER
W20260924 05:02:54.115111 21119 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c6f794a9b8c74d74a1bea28b8adb7d0b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 05:02:54.115211 21119 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 05:02:54.115307 21111 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:02:54.115509 21111 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:02:54.116545 21102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:54.110563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54719 (local address 127.0.0.1:37969)
0924 05:02:54.110747 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 05:02:54.110758 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 05:02:54.110796 (+ 38us) server_negotiation.cc:407] Connection header received
0924 05:02:54.111508 (+ 712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:02:54.111517 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:02:54.111579 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:02:54.111725 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:02:54.111910 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:54.113193 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:54.114290 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:54.115053 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:54.115161 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:54.115871 (+ 710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:02:54.115875 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:02:54.115875 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:02:54.116078 (+ 203us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:02:54.116114 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:02:54.116130 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:02:54.116195 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 05:02:54.116281 (+ 86us) server_negotiation.cc:299] Negotiation successful
0924 05:02:54.116343 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":77,"threads_started":1}
I20260924 05:02:54.116914 21111 catalog_manager.cc:1347] Generated new cluster ID: e516168d7efe49b6bf7e1dd05e157abc
I20260924 05:02:54.116959 21111 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:02:54.117820 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 05:02:54.125043 21102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:54.121679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42183 (local address 127.0.0.1:37969)
0924 05:02:54.121715 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 05:02:54.121717 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:02:54.121919 (+ 202us) server_negotiation.cc:407] Connection header received
0924 05:02:54.121963 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:02:54.121964 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:02:54.121975 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:02:54.122040 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:02:54.122148 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:54.122861 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:54.123800 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:54.124109 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:54.124173 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:54.124811 (+ 638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:02:54.124812 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:02:54.124813 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:02:54.124817 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:02:54.124833 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:02:54.124838 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:02:54.124886 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 05:02:54.124939 (+ 53us) server_negotiation.cc:299] Negotiation successful
0924 05:02:54.124962 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 05:02:54.129895 21111 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:02:54.130405 21111 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:02:54.137666 21111 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c6f794a9b8c74d74a1bea28b8adb7d0b: Generated new TSK 0
I20260924 05:02:54.137832 21111 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 05:02:54.144917 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 21030
2026-09-24T05:02:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1212 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790225929848825-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:02:54 dist-test-slave-70st krb5kdc[21129](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:02:54 dist-test-slave-70st krb5kdc[21129](info): set up 2 sockets
Sep 24 05:02:54 dist-test-slave-70st krb5kdc[21129](info): commencing operation
krb5kdc: starting...
W20260924 05:02:55.085180 392 mini_kdc.cc:121] Time spent starting KDC: real 0.924s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790225929848825-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790225929848825-392-0/krb5kdc/test-user.keytab.
Sep 24 05:02:55 dist-test-slave-70st krb5kdc[21129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:02:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:02:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790225929848825-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790225929848825-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 05:02:55.192082 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790225929848825-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790225929848825-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790225929848825-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43781
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790225929848825-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43781
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:02:55.269481 21142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:02:55.269676 21142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:02:55.269697 21142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:02:55.270810 21142 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:02:55.270851 21142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:02:55.270864 21142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:02:55.270877 21142 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:02:55.270900 21142 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:02:55.272367 21142 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790225929848825-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790225929848825-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43781
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790225929848825-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21142
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790225929848825-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:02:55.272580 21142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:02:55.272770 21142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:02:55.277292 21147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:02:55.277297 21150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:02:55.277475 21142 server_base.cc:1034] running on GCE node
W20260924 05:02:55.277303 21148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:02:55.277745 21142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:02:55.277995 21142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:02:55.279129 21142 hybrid_clock.cc:648] HybridClock initialized: now 1790226175279115 us; error 26 us; skew 500 ppm
Sep 24 05:02:55 dist-test-slave-70st krb5kdc[21129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226175, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:02:55.289078 21142 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 05:02:55.289525 21142 webserver.cc:460] Webserver started at http://127.0.0.1:39197/ using document root <none> and password file <none>
I20260924 05:02:55.289722 21142 fs_manager.cc:359] Metadata directory not provided
I20260924 05:02:55.289769 21142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:02:55.289888 21142 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:02:55.290897 21142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790225929848825-392-0/minicluster-data/master-0/data/instance:
uuid: "036a2a861d2e459dae8dc93300179c7e"
format_stamp: "Formatted at 2026-09-24 05:02:55 on dist-test-slave-70st"
server_key: "a67fe11782802736591b51c83bcf45e0"
server_key_iv: "8055a345558cef51a3b3ec6d1fe9799c"
server_key_version: "encryptionkey@0"
I20260924 05:02:55.291218 21142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790225929848825-392-0/minicluster-data/master-0/wal/instance:
uuid: "036a2a861d2e459dae8dc93300179c7e"
format_stamp: "Formatted at 2026-09-24 05:02:55 on dist-test-slave-70st"
server_key: "a67fe11782802736591b51c83bcf45e0"
server_key_iv: "8055a345558cef51a3b3ec6d1fe9799c"
server_key_version: "encryptionkey@0"
I20260924 05:02:55.292410 21142 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 05:02:55.293027 21157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:02:55.293186 21142 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 05:02:55.293267 21142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790225929848825-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790225929848825-392-0/minicluster-data/master-0/wal
uuid: "036a2a861d2e459dae8dc93300179c7e"
format_stamp: "Formatted at 2026-09-24 05:02:55 on dist-test-slave-70st"
server_key: "a67fe11782802736591b51c83bcf45e0"
server_key_iv: "8055a345558cef51a3b3ec6d1fe9799c"
server_key_version: "encryptionkey@0"
I20260924 05:02:55.293325 21142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790225929848825-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790225929848825-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790225929848825-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:02:55.296494 21142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:02:55.298213 21142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:02:55.298338 21142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:02:55.301847 21142 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43781
I20260924 05:02:55.301874 21209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43781 every 8 connection(s)
I20260924 05:02:55.302168 21142 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:02:55.303020 21210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:02:55.305056 21210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 036a2a861d2e459dae8dc93300179c7e: Bootstrap starting.
I20260924 05:02:55.305578 21210 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 036a2a861d2e459dae8dc93300179c7e: Neither blocks nor log segments found. Creating new log.
I20260924 05:02:55.305795 21210 log.cc:824] T 00000000000000000000000000000000 P 036a2a861d2e459dae8dc93300179c7e: Log is configured to *not* fsync() on all Append() calls
I20260924 05:02:55.306063 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 21142
I20260924 05:02:55.306164 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790225929848825-392-0/minicluster-data/master-0/wal/instance
I20260924 05:02:55.306331 392 external_mini_cluster.cc:1428] Setting key 8c55cb3da8aa0d1c73317be211e56fca
I20260924 05:02:55.306463 21210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 036a2a861d2e459dae8dc93300179c7e: No bootstrap required, opened a new log
I20260924 05:02:55.308420 21210 raft_consensus.cc:348] T 00000000000000000000000000000000 P 036a2a861d2e459dae8dc93300179c7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "036a2a861d2e459dae8dc93300179c7e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43781 } }
I20260924 05:02:55.308605 21210 raft_consensus.cc:374] T 00000000000000000000000000000000 P 036a2a861d2e459dae8dc93300179c7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:02:55.308670 21210 raft_consensus.cc:729] T 00000000000000000000000000000000 P 036a2a861d2e459dae8dc93300179c7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 036a2a861d2e459dae8dc93300179c7e, State: Initialized, Role: FOLLOWER
I20260924 05:02:55.308785 21210 consensus_queue.cc:260] T 00000000000000000000000000000000 P 036a2a861d2e459dae8dc93300179c7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "036a2a861d2e459dae8dc93300179c7e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43781 } }
I20260924 05:02:55.308852 21210 raft_consensus.cc:388] T 00000000000000000000000000000000 P 036a2a861d2e459dae8dc93300179c7e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:02:55.308881 21210 raft_consensus.cc:482] T 00000000000000000000000000000000 P 036a2a861d2e459dae8dc93300179c7e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:02:55.308905 21214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:55.306799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34329 (local address 127.0.0.1:43781)
0924 05:02:55.306971 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 05:02:55.306979 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 05:02:55.307003 (+ 24us) server_negotiation.cc:407] Connection header received
0924 05:02:55.307609 (+ 606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:02:55.307615 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:02:55.307639 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:02:55.307744 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:02:55.308306 (+ 562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:02:55.308309 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:02:55.308309 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:02:55.308312 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:02:55.308347 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:02:55.308356 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:02:55.308405 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 05:02:55.308483 (+ 78us) server_negotiation.cc:299] Negotiation successful
0924 05:02:55.308513 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":69,"threads_started":1}
I20260924 05:02:55.308923 21210 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 036a2a861d2e459dae8dc93300179c7e [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:02:55.309453 21210 raft_consensus.cc:504] T 00000000000000000000000000000000 P 036a2a861d2e459dae8dc93300179c7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "036a2a861d2e459dae8dc93300179c7e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43781 } }
I20260924 05:02:55.309551 21210 leader_election.cc:302] T 00000000000000000000000000000000 P 036a2a861d2e459dae8dc93300179c7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 036a2a861d2e459dae8dc93300179c7e; no voters:
I20260924 05:02:55.309690 21210 leader_election.cc:288] T 00000000000000000000000000000000 P 036a2a861d2e459dae8dc93300179c7e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:02:55.309715 21215 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 036a2a861d2e459dae8dc93300179c7e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:02:55.309767 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 05:02:55.309856 21215 raft_consensus.cc:686] T 00000000000000000000000000000000 P 036a2a861d2e459dae8dc93300179c7e [term 1 LEADER]: Becoming Leader. State: Replica: 036a2a861d2e459dae8dc93300179c7e, State: Running, Role: LEADER
I20260924 05:02:55.309975 21210 sys_catalog.cc:564] T 00000000000000000000000000000000 P 036a2a861d2e459dae8dc93300179c7e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:02:55.309986 21215 consensus_queue.cc:237] T 00000000000000000000000000000000 P 036a2a861d2e459dae8dc93300179c7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "036a2a861d2e459dae8dc93300179c7e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43781 } }
I20260924 05:02:55.310312 21216 sys_catalog.cc:455] T 00000000000000000000000000000000 P 036a2a861d2e459dae8dc93300179c7e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 036a2a861d2e459dae8dc93300179c7e. Latest consensus state: current_term: 1 leader_uuid: "036a2a861d2e459dae8dc93300179c7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "036a2a861d2e459dae8dc93300179c7e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43781 } } }
I20260924 05:02:55.310390 21216 sys_catalog.cc:458] T 00000000000000000000000000000000 P 036a2a861d2e459dae8dc93300179c7e [sys.catalog]: This master's current role is: LEADER
I20260924 05:02:55.310592 21217 sys_catalog.cc:455] T 00000000000000000000000000000000 P 036a2a861d2e459dae8dc93300179c7e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "036a2a861d2e459dae8dc93300179c7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "036a2a861d2e459dae8dc93300179c7e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43781 } } }
I20260924 05:02:55.310873 21217 sys_catalog.cc:458] T 00000000000000000000000000000000 P 036a2a861d2e459dae8dc93300179c7e [sys.catalog]: This master's current role is: LEADER
W20260924 05:02:55.311627 21235 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 036a2a861d2e459dae8dc93300179c7e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 05:02:55.311679 21235 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 05:02:55.311820 21221 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:02:55.311975 21221 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:02:55.312309 21214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:55.311206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44069 (local address 127.0.0.1:43781)
0924 05:02:55.311234 (+ 28us) server_negotiation.cc:206] Beginning negotiation
0924 05:02:55.311235 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:02:55.311387 (+ 152us) server_negotiation.cc:407] Connection header received
0924 05:02:55.311451 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:02:55.311452 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:02:55.311456 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:02:55.311499 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:02:55.312059 (+ 560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:02:55.312060 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:02:55.312060 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:02:55.312064 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:02:55.312077 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:02:55.312081 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:02:55.312118 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0924 05:02:55.312239 (+ 121us) server_negotiation.cc:299] Negotiation successful
0924 05:02:55.312250 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12}
I20260924 05:02:55.313323 21221 catalog_manager.cc:1347] Generated new cluster ID: b4ebaa09245b4b6f87f256e4271eb406
I20260924 05:02:55.313369 21221 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:02:55.321077 21221 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:02:55.321210 21221 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:02:55.337265 21221 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 036a2a861d2e459dae8dc93300179c7e: Generated new TSK 0
I20260924 05:02:55.337464 21221 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 05:02:55.347958 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 21142
2026-09-24T05:02:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1203 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790225929848825-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:02:55 dist-test-slave-70st krb5kdc[21241](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:02:55 dist-test-slave-70st krb5kdc[21241](info): set up 2 sockets
Sep 24 05:02:55 dist-test-slave-70st krb5kdc[21241](info): commencing operation
krb5kdc: starting...
W20260924 05:02:56.289180 392 mini_kdc.cc:121] Time spent starting KDC: real 0.924s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790225929848825-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790225929848825-392-0/krb5kdc/test-user.keytab.
Sep 24 05:02:56 dist-test-slave-70st krb5kdc[21241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226176, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:02:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:02:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790225929848825-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790225929848825-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 05:02:56.399384 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790225929848825-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790225929848825-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790225929848825-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34693
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790225929848825-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34693
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:02:56.477517 21254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:02:56.477723 21254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:02:56.477752 21254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:02:56.478920 21254 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:02:56.478962 21254 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:02:56.478981 21254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:02:56.478996 21254 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:02:56.479010 21254 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:02:56.480448 21254 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790225929848825-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790225929848825-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34693
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790225929848825-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34693
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21254
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790225929848825-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:02:56.480681 21254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:02:56.480890 21254 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:02:56.485409 21259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:02:56.485369 21260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:02:56.485442 21254 server_base.cc:1034] running on GCE node
W20260924 05:02:56.485387 21262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:02:56.485859 21254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:02:56.486124 21254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:02:56.487272 21254 hybrid_clock.cc:648] HybridClock initialized: now 1790226176487256 us; error 28 us; skew 500 ppm
Sep 24 05:02:56 dist-test-slave-70st krb5kdc[21241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226176, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:02:56.489593 21254 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 05:02:56.490013 21254 webserver.cc:460] Webserver started at http://127.0.0.1:41347/ using document root <none> and password file <none>
I20260924 05:02:56.490211 21254 fs_manager.cc:359] Metadata directory not provided
I20260924 05:02:56.490257 21254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:02:56.490386 21254 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:02:56.491381 21254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790225929848825-392-0/minicluster-data/master-0/data/instance:
uuid: "bbd3cb0103dd457bab4b1cbb0d4587d7"
format_stamp: "Formatted at 2026-09-24 05:02:56 on dist-test-slave-70st"
server_key: "831199c6e168be27740a9aa4086659ec"
server_key_iv: "42c88e6443b51e5dbdbaaa10f7415e5d"
server_key_version: "encryptionkey@0"
I20260924 05:02:56.491706 21254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790225929848825-392-0/minicluster-data/master-0/wal/instance:
uuid: "bbd3cb0103dd457bab4b1cbb0d4587d7"
format_stamp: "Formatted at 2026-09-24 05:02:56 on dist-test-slave-70st"
server_key: "831199c6e168be27740a9aa4086659ec"
server_key_iv: "42c88e6443b51e5dbdbaaa10f7415e5d"
server_key_version: "encryptionkey@0"
I20260924 05:02:56.492878 21254 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260924 05:02:56.493559 21269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:02:56.493708 21254 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:02:56.493788 21254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790225929848825-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790225929848825-392-0/minicluster-data/master-0/wal
uuid: "bbd3cb0103dd457bab4b1cbb0d4587d7"
format_stamp: "Formatted at 2026-09-24 05:02:56 on dist-test-slave-70st"
server_key: "831199c6e168be27740a9aa4086659ec"
server_key_iv: "42c88e6443b51e5dbdbaaa10f7415e5d"
server_key_version: "encryptionkey@0"
I20260924 05:02:56.493854 21254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790225929848825-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790225929848825-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790225929848825-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:02:56.509747 21254 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:02:56.511795 21254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:02:56.511926 21254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:02:56.515578 21254 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34693
I20260924 05:02:56.515618 21321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34693 every 8 connection(s)
I20260924 05:02:56.515926 21254 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:02:56.516700 21322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:02:56.518855 21322 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbd3cb0103dd457bab4b1cbb0d4587d7: Bootstrap starting.
I20260924 05:02:56.519362 21322 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bbd3cb0103dd457bab4b1cbb0d4587d7: Neither blocks nor log segments found. Creating new log.
I20260924 05:02:56.519585 21322 log.cc:824] T 00000000000000000000000000000000 P bbd3cb0103dd457bab4b1cbb0d4587d7: Log is configured to *not* fsync() on all Append() calls
I20260924 05:02:56.520167 21322 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbd3cb0103dd457bab4b1cbb0d4587d7: No bootstrap required, opened a new log
I20260924 05:02:56.521727 21322 raft_consensus.cc:348] T 00000000000000000000000000000000 P bbd3cb0103dd457bab4b1cbb0d4587d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbd3cb0103dd457bab4b1cbb0d4587d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34693 } }
I20260924 05:02:56.521863 21322 raft_consensus.cc:374] T 00000000000000000000000000000000 P bbd3cb0103dd457bab4b1cbb0d4587d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:02:56.521883 21322 raft_consensus.cc:729] T 00000000000000000000000000000000 P bbd3cb0103dd457bab4b1cbb0d4587d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbd3cb0103dd457bab4b1cbb0d4587d7, State: Initialized, Role: FOLLOWER
I20260924 05:02:56.521991 21322 consensus_queue.cc:260] T 00000000000000000000000000000000 P bbd3cb0103dd457bab4b1cbb0d4587d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbd3cb0103dd457bab4b1cbb0d4587d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34693 } }
I20260924 05:02:56.522063 21322 raft_consensus.cc:388] T 00000000000000000000000000000000 P bbd3cb0103dd457bab4b1cbb0d4587d7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:02:56.522089 21322 raft_consensus.cc:482] T 00000000000000000000000000000000 P bbd3cb0103dd457bab4b1cbb0d4587d7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:02:56.522122 21322 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bbd3cb0103dd457bab4b1cbb0d4587d7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:02:56.522569 21322 raft_consensus.cc:504] T 00000000000000000000000000000000 P bbd3cb0103dd457bab4b1cbb0d4587d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbd3cb0103dd457bab4b1cbb0d4587d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34693 } }
I20260924 05:02:56.522650 21322 leader_election.cc:302] T 00000000000000000000000000000000 P bbd3cb0103dd457bab4b1cbb0d4587d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bbd3cb0103dd457bab4b1cbb0d4587d7; no voters:
I20260924 05:02:56.522766 21322 leader_election.cc:288] T 00000000000000000000000000000000 P bbd3cb0103dd457bab4b1cbb0d4587d7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:02:56.522827 21325 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bbd3cb0103dd457bab4b1cbb0d4587d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:02:56.522959 21325 raft_consensus.cc:686] T 00000000000000000000000000000000 P bbd3cb0103dd457bab4b1cbb0d4587d7 [term 1 LEADER]: Becoming Leader. State: Replica: bbd3cb0103dd457bab4b1cbb0d4587d7, State: Running, Role: LEADER
I20260924 05:02:56.523010 21322 sys_catalog.cc:564] T 00000000000000000000000000000000 P bbd3cb0103dd457bab4b1cbb0d4587d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:02:56.523037 21325 consensus_queue.cc:237] T 00000000000000000000000000000000 P bbd3cb0103dd457bab4b1cbb0d4587d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbd3cb0103dd457bab4b1cbb0d4587d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34693 } }
I20260924 05:02:56.523504 21326 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbd3cb0103dd457bab4b1cbb0d4587d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bbd3cb0103dd457bab4b1cbb0d4587d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbd3cb0103dd457bab4b1cbb0d4587d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34693 } } }
I20260924 05:02:56.523546 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 21254
I20260924 05:02:56.523671 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790225929848825-392-0/minicluster-data/master-0/wal/instance
I20260924 05:02:56.523723 21326 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbd3cb0103dd457bab4b1cbb0d4587d7 [sys.catalog]: This master's current role is: LEADER
I20260924 05:02:56.523865 392 external_mini_cluster.cc:1428] Setting key a93bb3eccb42940d5e20b08e224c73c6
I20260924 05:02:56.523661 21327 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbd3cb0103dd457bab4b1cbb0d4587d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bbd3cb0103dd457bab4b1cbb0d4587d7. Latest consensus state: current_term: 1 leader_uuid: "bbd3cb0103dd457bab4b1cbb0d4587d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbd3cb0103dd457bab4b1cbb0d4587d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34693 } } }
I20260924 05:02:56.524031 21327 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbd3cb0103dd457bab4b1cbb0d4587d7 [sys.catalog]: This master's current role is: LEADER
I20260924 05:02:56.524091 21334 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:02:56.524586 21334 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:02:56.526012 21334 catalog_manager.cc:1347] Generated new cluster ID: 17ad2031f30c4285b6bccbaa8dd20ab0
I20260924 05:02:56.526062 21334 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 05:02:56 dist-test-slave-70st krb5kdc[21241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226176, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 05:02:56.535194 21343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:56.524416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.192:58707 (local address 127.0.0.1:34693)
0924 05:02:56.524703 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0924 05:02:56.524713 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 05:02:56.524744 (+ 31us) server_negotiation.cc:407] Connection header received
0924 05:02:56.524878 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:02:56.524884 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:02:56.524939 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:02:56.525089 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:02:56.525851 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:56.527216 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:56.528243 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:56.528865 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:56.528987 (+ 122us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:56.531642 (+ 2655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:02:56.531646 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:02:56.531647 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:02:56.531658 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:02:56.533535 (+ 1877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:02:56.533983 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:02:56.533987 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:02:56.533987 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:02:56.534083 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:02:56.534285 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:02:56.534285 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:02:56.534286 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:02:56.534567 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:02:56.534707 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 05:02:56.534867 (+ 160us) server_negotiation.cc:299] Negotiation successful
0924 05:02:56.534960 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":79,"threads_started":1}
I20260924 05:02:56.536355 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 05:02:56.544793 21343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:56.537797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.192:53179 (local address 127.0.0.1:34693)
0924 05:02:56.537834 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 05:02:56.537836 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:02:56.537981 (+ 145us) server_negotiation.cc:407] Connection header received
0924 05:02:56.538063 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:02:56.538064 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:02:56.538074 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:02:56.538118 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:02:56.538766 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:56.539518 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:56.540493 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:56.540802 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:56.540860 (+ 58us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:56.542199 (+ 1339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:02:56.542200 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:02:56.542200 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:02:56.542206 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:02:56.543629 (+ 1423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:02:56.544038 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:02:56.544039 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:02:56.544039 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:02:56.544092 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:02:56.544292 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:02:56.544293 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:02:56.544293 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:02:56.544407 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:02:56.544501 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 05:02:56.544688 (+ 187us) server_negotiation.cc:299] Negotiation successful
0924 05:02:56.544736 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 05:02:56.545079 21334 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:02:56.545775 21334 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:02:56.554282 21334 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bbd3cb0103dd457bab4b1cbb0d4587d7: Generated new TSK 0
I20260924 05:02:56.554461 21334 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 05:02:56.558203 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 21254
W20260924 05:02:56.561940 21239 connection.cc:535] client connection to 127.0.0.1:34693 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34693): Connection reset by peer (error 104)
2026-09-24T05:02:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1210 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790225929848825-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:02:56 dist-test-slave-70st krb5kdc[21353](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:02:56 dist-test-slave-70st krb5kdc[21353](info): set up 2 sockets
Sep 24 05:02:56 dist-test-slave-70st krb5kdc[21353](info): commencing operation
krb5kdc: starting...
W20260924 05:02:57.497226 392 mini_kdc.cc:121] Time spent starting KDC: real 0.922s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790225929848825-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790225929848825-392-0/krb5kdc/test-user.keytab.
Sep 24 05:02:57 dist-test-slave-70st krb5kdc[21353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226177, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:02:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:02:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790225929848825-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790225929848825-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 05:02:57.608160 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790225929848825-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790225929848825-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790225929848825-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38171
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790225929848825-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38171
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:02:57.689445 21366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:02:57.689648 21366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:02:57.689680 21366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:02:57.690824 21366 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:02:57.690867 21366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:02:57.690886 21366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:02:57.690901 21366 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:02:57.690917 21366 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:02:57.692363 21366 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790225929848825-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790225929848825-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38171
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790225929848825-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38171
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21366
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790225929848825-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:02:57.692592 21366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:02:57.692793 21366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:02:57.697367 21371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:02:57.697476 21372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:02:57.697398 21374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:02:57.697621 21366 server_base.cc:1034] running on GCE node
I20260924 05:02:57.697826 21366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:02:57.698089 21366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:02:57.699229 21366 hybrid_clock.cc:648] HybridClock initialized: now 1790226177699216 us; error 29 us; skew 500 ppm
Sep 24 05:02:57 dist-test-slave-70st krb5kdc[21353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226177, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:02:57.701553 21366 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 05:02:57.701977 21366 webserver.cc:460] Webserver started at http://127.0.0.1:37693/ using document root <none> and password file <none>
I20260924 05:02:57.702163 21366 fs_manager.cc:359] Metadata directory not provided
I20260924 05:02:57.702201 21366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:02:57.702288 21366 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:02:57.703256 21366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790225929848825-392-0/minicluster-data/master-0/data/instance:
uuid: "1f85ff77fdb946f6b5396dcd842bbb53"
format_stamp: "Formatted at 2026-09-24 05:02:57 on dist-test-slave-70st"
server_key: "a3d7d0ec5e1c9191f59a4af40e3ab855"
server_key_iv: "c5afac22bc81b4f29bc85c6596b45206"
server_key_version: "encryptionkey@0"
I20260924 05:02:57.703529 21366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790225929848825-392-0/minicluster-data/master-0/wal/instance:
uuid: "1f85ff77fdb946f6b5396dcd842bbb53"
format_stamp: "Formatted at 2026-09-24 05:02:57 on dist-test-slave-70st"
server_key: "a3d7d0ec5e1c9191f59a4af40e3ab855"
server_key_iv: "c5afac22bc81b4f29bc85c6596b45206"
server_key_version: "encryptionkey@0"
I20260924 05:02:57.704681 21366 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 05:02:57.705286 21381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:02:57.705466 21366 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:02:57.705552 21366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790225929848825-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790225929848825-392-0/minicluster-data/master-0/wal
uuid: "1f85ff77fdb946f6b5396dcd842bbb53"
format_stamp: "Formatted at 2026-09-24 05:02:57 on dist-test-slave-70st"
server_key: "a3d7d0ec5e1c9191f59a4af40e3ab855"
server_key_iv: "c5afac22bc81b4f29bc85c6596b45206"
server_key_version: "encryptionkey@0"
I20260924 05:02:57.705614 21366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790225929848825-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790225929848825-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790225929848825-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:02:57.718127 21366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:02:57.719937 21366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:02:57.720072 21366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:02:57.723644 21366 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38171
I20260924 05:02:57.723681 21433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38171 every 8 connection(s)
I20260924 05:02:57.724002 21366 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:02:57.724815 21434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:02:57.727042 21434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f85ff77fdb946f6b5396dcd842bbb53: Bootstrap starting.
I20260924 05:02:57.727541 21434 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1f85ff77fdb946f6b5396dcd842bbb53: Neither blocks nor log segments found. Creating new log.
I20260924 05:02:57.727764 21434 log.cc:824] T 00000000000000000000000000000000 P 1f85ff77fdb946f6b5396dcd842bbb53: Log is configured to *not* fsync() on all Append() calls
I20260924 05:02:57.728364 21434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f85ff77fdb946f6b5396dcd842bbb53: No bootstrap required, opened a new log
I20260924 05:02:57.729943 21434 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1f85ff77fdb946f6b5396dcd842bbb53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f85ff77fdb946f6b5396dcd842bbb53" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38171 } }
I20260924 05:02:57.730075 21434 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1f85ff77fdb946f6b5396dcd842bbb53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:02:57.730103 21434 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1f85ff77fdb946f6b5396dcd842bbb53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f85ff77fdb946f6b5396dcd842bbb53, State: Initialized, Role: FOLLOWER
I20260924 05:02:57.730213 21434 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1f85ff77fdb946f6b5396dcd842bbb53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f85ff77fdb946f6b5396dcd842bbb53" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38171 } }
I20260924 05:02:57.730288 21434 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1f85ff77fdb946f6b5396dcd842bbb53 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:02:57.730315 21434 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1f85ff77fdb946f6b5396dcd842bbb53 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:02:57.730347 21434 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1f85ff77fdb946f6b5396dcd842bbb53 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:02:57.730779 21434 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1f85ff77fdb946f6b5396dcd842bbb53 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f85ff77fdb946f6b5396dcd842bbb53" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38171 } }
I20260924 05:02:57.730859 21434 leader_election.cc:302] T 00000000000000000000000000000000 P 1f85ff77fdb946f6b5396dcd842bbb53 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1f85ff77fdb946f6b5396dcd842bbb53; no voters:
I20260924 05:02:57.730973 21434 leader_election.cc:288] T 00000000000000000000000000000000 P 1f85ff77fdb946f6b5396dcd842bbb53 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:02:57.731019 21437 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1f85ff77fdb946f6b5396dcd842bbb53 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:02:57.731177 21437 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1f85ff77fdb946f6b5396dcd842bbb53 [term 1 LEADER]: Becoming Leader. State: Replica: 1f85ff77fdb946f6b5396dcd842bbb53, State: Running, Role: LEADER
I20260924 05:02:57.731175 21434 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1f85ff77fdb946f6b5396dcd842bbb53 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:02:57.731302 21437 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1f85ff77fdb946f6b5396dcd842bbb53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f85ff77fdb946f6b5396dcd842bbb53" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38171 } }
I20260924 05:02:57.731686 21438 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f85ff77fdb946f6b5396dcd842bbb53 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1f85ff77fdb946f6b5396dcd842bbb53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f85ff77fdb946f6b5396dcd842bbb53" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38171 } } }
I20260924 05:02:57.731812 21438 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f85ff77fdb946f6b5396dcd842bbb53 [sys.catalog]: This master's current role is: LEADER
I20260924 05:02:57.731799 21439 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f85ff77fdb946f6b5396dcd842bbb53 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1f85ff77fdb946f6b5396dcd842bbb53. Latest consensus state: current_term: 1 leader_uuid: "1f85ff77fdb946f6b5396dcd842bbb53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f85ff77fdb946f6b5396dcd842bbb53" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38171 } } }
I20260924 05:02:57.731974 21439 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f85ff77fdb946f6b5396dcd842bbb53 [sys.catalog]: This master's current role is: LEADER
I20260924 05:02:57.732309 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 21366
I20260924 05:02:57.732366 21446 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:02:57.732426 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790225929848825-392-0/minicluster-data/master-0/wal/instance
I20260924 05:02:57.732589 392 external_mini_cluster.cc:1428] Setting key 89fdfac67436bbbbdfb060de2410927f
I20260924 05:02:57.732940 21446 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:02:57.734352 21446 catalog_manager.cc:1347] Generated new cluster ID: bee26f1bdc8747f78cd79d658b856622
I20260924 05:02:57.734400 21446 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:02:57.738679 21456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:57.733214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.192:42095 (local address 127.0.0.1:38171)
0924 05:02:57.733520 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0924 05:02:57.733529 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 05:02:57.733563 (+ 34us) server_negotiation.cc:407] Connection header received
0924 05:02:57.733672 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:02:57.733679 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:02:57.733737 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:02:57.733887 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:02:57.734111 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:57.735474 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:57.736575 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:57.737214 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:57.737342 (+ 128us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:57.738015 (+ 673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:02:57.738020 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:02:57.738020 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:02:57.738224 (+ 204us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:02:57.738267 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:02:57.738282 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:02:57.738364 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 05:02:57.738444 (+ 80us) server_negotiation.cc:299] Negotiation successful
0924 05:02:57.738474 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":61,"threads_started":1}
I20260924 05:02:57.739903 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 05:02:57.741361 21446 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:02:57.741834 21446 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:02:57.743947 21446 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1f85ff77fdb946f6b5396dcd842bbb53: Generated new TSK 0
I20260924 05:02:57.744102 21446 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 05:02:57.746783 21456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:57.741308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.192:48049 (local address 127.0.0.1:38171)
0924 05:02:57.741473 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 05:02:57.741474 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:02:57.743863 (+ 2389us) server_negotiation.cc:407] Connection header received
0924 05:02:57.743882 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:02:57.743883 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:02:57.743894 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:02:57.743941 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:02:57.744109 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:57.744636 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:57.745717 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:02:57.745912 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:02:57.745961 (+ 49us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:02:57.746568 (+ 607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:02:57.746569 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:02:57.746569 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:02:57.746573 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:02:57.746587 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:02:57.746591 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:02:57.746633 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0924 05:02:57.746707 (+ 74us) server_negotiation.cc:299] Negotiation successful
0924 05:02:57.746718 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148}
I20260924 05:02:57.748677 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 21366
W20260924 05:02:57.752174 21351 connection.cc:535] client connection to 127.0.0.1:38171 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38171): Connection reset by peer (error 104)
2026-09-24T05:02:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1190 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790225929848825-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:02:57 dist-test-slave-70st krb5kdc[21465](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:02:57 dist-test-slave-70st krb5kdc[21465](info): set up 2 sockets
Sep 24 05:02:57 dist-test-slave-70st krb5kdc[21465](info): commencing operation
krb5kdc: starting...
W20260924 05:02:58.689137 392 mini_kdc.cc:121] Time spent starting KDC: real 0.924s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790225929848825-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790225929848825-392-0/krb5kdc/test-user.keytab.
Sep 24 05:02:58 dist-test-slave-70st krb5kdc[21465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226178, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:02:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:02:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790225929848825-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790225929848825-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 05:02:58.797905 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790225929848825-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790225929848825-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790225929848825-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41653
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790225929848825-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41653
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:02:58.875257 21478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:02:58.875455 21478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:02:58.875476 21478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:02:58.876590 21478 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:02:58.876629 21478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:02:58.876643 21478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:02:58.876667 21478 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:02:58.876688 21478 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:02:58.878113 21478 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790225929848825-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790225929848825-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41653
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790225929848825-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21478
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790225929848825-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:02:58.878334 21478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:02:58.878531 21478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:02:58.883031 21486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:02:58.883064 21478 server_base.cc:1034] running on GCE node
W20260924 05:02:58.883006 21484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:02:58.883019 21483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:02:58.883397 21478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:02:58.883648 21478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:02:58.884783 21478 hybrid_clock.cc:648] HybridClock initialized: now 1790226178884768 us; error 29 us; skew 500 ppm
Sep 24 05:02:58 dist-test-slave-70st krb5kdc[21465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226178, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:02:58.887029 21478 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 05:02:58.887452 21478 webserver.cc:460] Webserver started at http://127.0.0.1:41073/ using document root <none> and password file <none>
I20260924 05:02:58.887637 21478 fs_manager.cc:359] Metadata directory not provided
I20260924 05:02:58.887682 21478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:02:58.887799 21478 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:02:58.888749 21478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790225929848825-392-0/minicluster-data/master-0/data/instance:
uuid: "fc798b0511e7484f95a5cd14288195cf"
format_stamp: "Formatted at 2026-09-24 05:02:58 on dist-test-slave-70st"
server_key: "6ccf14aa7537e8fab95c93a5e78235fd"
server_key_iv: "07fa15f1993f1528799be0b6e1fa1b49"
server_key_version: "encryptionkey@0"
I20260924 05:02:58.889070 21478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790225929848825-392-0/minicluster-data/master-0/wal/instance:
uuid: "fc798b0511e7484f95a5cd14288195cf"
format_stamp: "Formatted at 2026-09-24 05:02:58 on dist-test-slave-70st"
server_key: "6ccf14aa7537e8fab95c93a5e78235fd"
server_key_iv: "07fa15f1993f1528799be0b6e1fa1b49"
server_key_version: "encryptionkey@0"
I20260924 05:02:58.890254 21478 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 05:02:58.890872 21493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:02:58.891012 21478 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 05:02:58.891085 21478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790225929848825-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790225929848825-392-0/minicluster-data/master-0/wal
uuid: "fc798b0511e7484f95a5cd14288195cf"
format_stamp: "Formatted at 2026-09-24 05:02:58 on dist-test-slave-70st"
server_key: "6ccf14aa7537e8fab95c93a5e78235fd"
server_key_iv: "07fa15f1993f1528799be0b6e1fa1b49"
server_key_version: "encryptionkey@0"
I20260924 05:02:58.891141 21478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790225929848825-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790225929848825-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790225929848825-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:02:58.894145 21478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:02:58.895699 21478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:02:58.895812 21478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:02:58.899255 21478 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41653
I20260924 05:02:58.899281 21545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41653 every 8 connection(s)
I20260924 05:02:58.899547 21478 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:02:58.900372 21546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:02:58.901633 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 21478
I20260924 05:02:58.901743 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790225929848825-392-0/minicluster-data/master-0/wal/instance
I20260924 05:02:58.901894 392 external_mini_cluster.cc:1428] Setting key 46e53e805f1dc2d09376b98fcda81fd7
I20260924 05:02:58.902747 21546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc798b0511e7484f95a5cd14288195cf: Bootstrap starting.
I20260924 05:02:58.903385 21546 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc798b0511e7484f95a5cd14288195cf: Neither blocks nor log segments found. Creating new log.
I20260924 05:02:58.903714 21546 log.cc:824] T 00000000000000000000000000000000 P fc798b0511e7484f95a5cd14288195cf: Log is configured to *not* fsync() on all Append() calls
I20260924 05:02:58.904177 21549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:58.902307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.192:55731 (local address 127.0.0.1:41653)
0924 05:02:58.902486 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 05:02:58.902494 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 05:02:58.902525 (+ 31us) server_negotiation.cc:407] Connection header received
0924 05:02:58.903010 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:02:58.903019 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:02:58.903055 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:02:58.903205 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:02:58.903782 (+ 577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:02:58.903785 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:02:58.903786 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:02:58.903791 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:02:58.903840 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:02:58.903854 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:02:58.903928 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 05:02:58.903970 (+ 42us) server_negotiation.cc:299] Negotiation successful
0924 05:02:58.904010 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":70,"threads_started":1}
I20260924 05:02:58.904381 21546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc798b0511e7484f95a5cd14288195cf: No bootstrap required, opened a new log
I20260924 05:02:58.905019 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 05:02:58.906401 21546 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc798b0511e7484f95a5cd14288195cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc798b0511e7484f95a5cd14288195cf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41653 } }
I20260924 05:02:58.906524 21546 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc798b0511e7484f95a5cd14288195cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:02:58.906554 21546 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc798b0511e7484f95a5cd14288195cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc798b0511e7484f95a5cd14288195cf, State: Initialized, Role: FOLLOWER
I20260924 05:02:58.906663 21546 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc798b0511e7484f95a5cd14288195cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc798b0511e7484f95a5cd14288195cf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41653 } }
I20260924 05:02:58.906741 21546 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc798b0511e7484f95a5cd14288195cf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:02:58.906801 21546 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc798b0511e7484f95a5cd14288195cf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:02:58.906853 21546 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc798b0511e7484f95a5cd14288195cf [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:02:58.907147 21549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:02:58.906271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.192:49413 (local address 127.0.0.1:41653)
0924 05:02:58.906321 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 05:02:58.906323 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:02:58.906424 (+ 101us) server_negotiation.cc:407] Connection header received
0924 05:02:58.906470 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:02:58.906470 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:02:58.906475 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:02:58.906518 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:02:58.906969 (+ 451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:02:58.906970 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:02:58.906971 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:02:58.906975 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:02:58.906986 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:02:58.906989 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:02:58.907033 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 05:02:58.907080 (+ 47us) server_negotiation.cc:299] Negotiation successful
0924 05:02:58.907091 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 05:02:58.907390 21546 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc798b0511e7484f95a5cd14288195cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc798b0511e7484f95a5cd14288195cf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41653 } }
I20260924 05:02:58.907483 21546 leader_election.cc:302] T 00000000000000000000000000000000 P fc798b0511e7484f95a5cd14288195cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc798b0511e7484f95a5cd14288195cf; no voters:
I20260924 05:02:58.907626 21546 leader_election.cc:288] T 00000000000000000000000000000000 P fc798b0511e7484f95a5cd14288195cf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:02:58.907653 21556 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc798b0511e7484f95a5cd14288195cf [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:02:58.907781 21556 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc798b0511e7484f95a5cd14288195cf [term 1 LEADER]: Becoming Leader. State: Replica: fc798b0511e7484f95a5cd14288195cf, State: Running, Role: LEADER
I20260924 05:02:58.907831 21546 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc798b0511e7484f95a5cd14288195cf [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:02:58.907855 21556 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc798b0511e7484f95a5cd14288195cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc798b0511e7484f95a5cd14288195cf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41653 } }
I20260924 05:02:58.908138 21556 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc798b0511e7484f95a5cd14288195cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc798b0511e7484f95a5cd14288195cf. Latest consensus state: current_term: 1 leader_uuid: "fc798b0511e7484f95a5cd14288195cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc798b0511e7484f95a5cd14288195cf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41653 } } }
I20260924 05:02:58.908149 21558 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc798b0511e7484f95a5cd14288195cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fc798b0511e7484f95a5cd14288195cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc798b0511e7484f95a5cd14288195cf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41653 } } }
I20260924 05:02:58.908325 21558 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc798b0511e7484f95a5cd14288195cf [sys.catalog]: This master's current role is: LEADER
I20260924 05:02:58.908622 21556 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc798b0511e7484f95a5cd14288195cf [sys.catalog]: This master's current role is: LEADER
I20260924 05:02:58.908741 21565 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:02:58.909170 21565 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:02:58.910624 21565 catalog_manager.cc:1347] Generated new cluster ID: 690b59fd19c147cfb9ac29270c441851
I20260924 05:02:58.910666 21565 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:02:58.935986 21565 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:02:58.936148 21565 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:02:58.941376 21565 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fc798b0511e7484f95a5cd14288195cf: Generated new TSK 0
I20260924 05:02:58.941558 21565 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 05:02:58.950721 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 21478
2026-09-24T05:02:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1202 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790225929848825-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:02:58 dist-test-slave-70st krb5kdc[21577](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:02:58 dist-test-slave-70st krb5kdc[21577](info): set up 2 sockets
Sep 24 05:02:58 dist-test-slave-70st krb5kdc[21577](info): commencing operation
krb5kdc: starting...
W20260924 05:02:59.893172 392 mini_kdc.cc:121] Time spent starting KDC: real 0.926s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790225929848825-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790225929848825-392-0/krb5kdc/test-user.keytab.
Sep 24 05:02:59 dist-test-slave-70st krb5kdc[21577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:02:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:02:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790225929848825-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790225929848825-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 05:03:00.011929 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790225929848825-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790225929848825-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790225929848825-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46201
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790225929848825-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46201
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:03:00.090583 21590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:03:00.090786 21590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:03:00.090807 21590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:03:00.091922 21590 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:03:00.091964 21590 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:03:00.091977 21590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:03:00.092000 21590 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:03:00.092015 21590 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:03:00.093487 21590 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790225929848825-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790225929848825-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46201
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790225929848825-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46201
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21590
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790225929848825-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:03:00.093694 21590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:03:00.093901 21590 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:03:00.098552 21595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:03:00.098549 21598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:03:00.098551 21596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:03:00.098654 21590 server_base.cc:1034] running on GCE node
I20260924 05:03:00.098982 21590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:03:00.099226 21590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:03:00.100384 21590 hybrid_clock.cc:648] HybridClock initialized: now 1790226180100370 us; error 29 us; skew 500 ppm
Sep 24 05:03:00 dist-test-slave-70st krb5kdc[21577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226180, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:03:00.102835 21590 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 05:03:00.103215 21590 webserver.cc:460] Webserver started at http://127.0.0.1:46255/ using document root <none> and password file <none>
I20260924 05:03:00.103403 21590 fs_manager.cc:359] Metadata directory not provided
I20260924 05:03:00.103444 21590 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:03:00.103530 21590 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:03:00.104487 21590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790225929848825-392-0/minicluster-data/master-0/data/instance:
uuid: "370b374f2e4c4da58938efa41e71d9de"
format_stamp: "Formatted at 2026-09-24 05:03:00 on dist-test-slave-70st"
server_key: "2557ed5f1997c6503c1757e8d5942475"
server_key_iv: "5c7dbcb8f046edaf1634e5ff0676d8f2"
server_key_version: "encryptionkey@0"
I20260924 05:03:00.104764 21590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790225929848825-392-0/minicluster-data/master-0/wal/instance:
uuid: "370b374f2e4c4da58938efa41e71d9de"
format_stamp: "Formatted at 2026-09-24 05:03:00 on dist-test-slave-70st"
server_key: "2557ed5f1997c6503c1757e8d5942475"
server_key_iv: "5c7dbcb8f046edaf1634e5ff0676d8f2"
server_key_version: "encryptionkey@0"
I20260924 05:03:00.105986 21590 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 05:03:00.106601 21605 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:03:00.106735 21590 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:03:00.106818 21590 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790225929848825-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790225929848825-392-0/minicluster-data/master-0/wal
uuid: "370b374f2e4c4da58938efa41e71d9de"
format_stamp: "Formatted at 2026-09-24 05:03:00 on dist-test-slave-70st"
server_key: "2557ed5f1997c6503c1757e8d5942475"
server_key_iv: "5c7dbcb8f046edaf1634e5ff0676d8f2"
server_key_version: "encryptionkey@0"
I20260924 05:03:00.106880 21590 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790225929848825-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790225929848825-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790225929848825-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:03:00.128680 21590 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:03:00.130808 21590 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:03:00.130927 21590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:03:00.134545 21590 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46201
I20260924 05:03:00.134591 21657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46201 every 8 connection(s)
I20260924 05:03:00.134907 21590 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:03:00.135703 21658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:03:00.136243 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 21590
I20260924 05:03:00.136358 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790225929848825-392-0/minicluster-data/master-0/wal/instance
I20260924 05:03:00.136512 392 external_mini_cluster.cc:1428] Setting key 0f7dc77533bdec7a163d7dc2ffbe0e5f
I20260924 05:03:00.138329 21658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 370b374f2e4c4da58938efa41e71d9de: Bootstrap starting.
I20260924 05:03:00.138891 21658 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 370b374f2e4c4da58938efa41e71d9de: Neither blocks nor log segments found. Creating new log.
I20260924 05:03:00.139140 21658 log.cc:824] T 00000000000000000000000000000000 P 370b374f2e4c4da58938efa41e71d9de: Log is configured to *not* fsync() on all Append() calls
I20260924 05:03:00.139726 21658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 370b374f2e4c4da58938efa41e71d9de: No bootstrap required, opened a new log
I20260924 05:03:00.141389 21658 raft_consensus.cc:348] T 00000000000000000000000000000000 P 370b374f2e4c4da58938efa41e71d9de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "370b374f2e4c4da58938efa41e71d9de" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46201 } }
I20260924 05:03:00.141552 21658 raft_consensus.cc:374] T 00000000000000000000000000000000 P 370b374f2e4c4da58938efa41e71d9de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:03:00.141597 21658 raft_consensus.cc:729] T 00000000000000000000000000000000 P 370b374f2e4c4da58938efa41e71d9de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 370b374f2e4c4da58938efa41e71d9de, State: Initialized, Role: FOLLOWER
I20260924 05:03:00.141713 21658 consensus_queue.cc:260] T 00000000000000000000000000000000 P 370b374f2e4c4da58938efa41e71d9de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "370b374f2e4c4da58938efa41e71d9de" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46201 } }
I20260924 05:03:00.141772 21658 raft_consensus.cc:388] T 00000000000000000000000000000000 P 370b374f2e4c4da58938efa41e71d9de [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:03:00.141800 21658 raft_consensus.cc:482] T 00000000000000000000000000000000 P 370b374f2e4c4da58938efa41e71d9de [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:03:00.141844 21658 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 370b374f2e4c4da58938efa41e71d9de [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:03:00.142311 21658 raft_consensus.cc:504] T 00000000000000000000000000000000 P 370b374f2e4c4da58938efa41e71d9de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "370b374f2e4c4da58938efa41e71d9de" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46201 } }
I20260924 05:03:00.142410 21658 leader_election.cc:302] T 00000000000000000000000000000000 P 370b374f2e4c4da58938efa41e71d9de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 370b374f2e4c4da58938efa41e71d9de; no voters:
I20260924 05:03:00.142565 21658 leader_election.cc:288] T 00000000000000000000000000000000 P 370b374f2e4c4da58938efa41e71d9de [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:03:00.142606 21663 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 370b374f2e4c4da58938efa41e71d9de [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:03:00.142755 21663 raft_consensus.cc:686] T 00000000000000000000000000000000 P 370b374f2e4c4da58938efa41e71d9de [term 1 LEADER]: Becoming Leader. State: Replica: 370b374f2e4c4da58938efa41e71d9de, State: Running, Role: LEADER
I20260924 05:03:00.142815 21658 sys_catalog.cc:564] T 00000000000000000000000000000000 P 370b374f2e4c4da58938efa41e71d9de [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:03:00.142894 21663 consensus_queue.cc:237] T 00000000000000000000000000000000 P 370b374f2e4c4da58938efa41e71d9de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "370b374f2e4c4da58938efa41e71d9de" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46201 } }
Sep 24 05:03:00 dist-test-slave-70st krb5kdc[21577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 05:03:00.143193 21663 sys_catalog.cc:455] T 00000000000000000000000000000000 P 370b374f2e4c4da58938efa41e71d9de [sys.catalog]: SysCatalogTable state changed. Reason: New leader 370b374f2e4c4da58938efa41e71d9de. Latest consensus state: current_term: 1 leader_uuid: "370b374f2e4c4da58938efa41e71d9de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "370b374f2e4c4da58938efa41e71d9de" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46201 } } }
I20260924 05:03:00.143329 21663 sys_catalog.cc:458] T 00000000000000000000000000000000 P 370b374f2e4c4da58938efa41e71d9de [sys.catalog]: This master's current role is: LEADER
I20260924 05:03:00.143553 21672 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:03:00.144198 21672 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:03:00.144068 21664 sys_catalog.cc:455] T 00000000000000000000000000000000 P 370b374f2e4c4da58938efa41e71d9de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "370b374f2e4c4da58938efa41e71d9de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "370b374f2e4c4da58938efa41e71d9de" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46201 } } }
I20260924 05:03:00.145548 21664 sys_catalog.cc:458] T 00000000000000000000000000000000 P 370b374f2e4c4da58938efa41e71d9de [sys.catalog]: This master's current role is: LEADER
I20260924 05:03:00.147274 21672 catalog_manager.cc:1347] Generated new cluster ID: dd6701ae833a4ccd85e551f940ca80b3
I20260924 05:03:00.147370 21672 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:03:00.147743 21661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:00.137022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.192:35037 (local address 127.0.0.1:46201)
0924 05:03:00.137200 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:00.137208 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:00.137268 (+ 60us) server_negotiation.cc:407] Connection header received
0924 05:03:00.137732 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:00.137739 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:00.137807 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:00.137937 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:03:00.138601 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:00.139588 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:00.140386 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:00.140837 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:00.140938 (+ 101us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:00.143559 (+ 2621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:03:00.143564 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:03:00.143565 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:03:00.143576 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:03:00.145348 (+ 1772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:00.146410 (+ 1062us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:00.146413 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:00.146413 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:00.146505 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:00.146781 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:00.146781 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:00.146782 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:00.147048 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:03:00.147212 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0924 05:03:00.147427 (+ 215us) server_negotiation.cc:299] Negotiation successful
0924 05:03:00.147533 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":77,"threads_started":1}
I20260924 05:03:00.148670 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 05:03:00.157166 21661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:00.150242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.192:34859 (local address 127.0.0.1:46201)
0924 05:03:00.150279 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:00.150281 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:00.150320 (+ 39us) server_negotiation.cc:407] Connection header received
0924 05:03:00.150390 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:00.150392 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:00.150400 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:00.150457 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:03:00.151028 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:00.151756 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:00.152703 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:00.153003 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:00.153067 (+ 64us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:00.154500 (+ 1433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:03:00.154501 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:03:00.154502 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:03:00.154508 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:03:00.156028 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:00.156421 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:00.156422 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:00.156423 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:00.156478 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:00.156675 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:00.156675 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:00.156676 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:00.156794 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:03:00.156903 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 05:03:00.157016 (+ 113us) server_negotiation.cc:299] Negotiation successful
0924 05:03:00.157074 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 05:03:00.160861 21672 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:03:00.161465 21672 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:03:00.175532 21672 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 370b374f2e4c4da58938efa41e71d9de: Generated new TSK 0
I20260924 05:03:00.175714 21672 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 05:03:00.181630 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 21590
W20260924 05:03:00.185343 21575 connection.cc:535] client connection to 127.0.0.1:46201 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46201): Connection reset by peer (error 104)
2026-09-24T05:03:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1230 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790225929848825-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:03:00 dist-test-slave-70st krb5kdc[21689](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:03:00 dist-test-slave-70st krb5kdc[21689](info): set up 2 sockets
Sep 24 05:03:00 dist-test-slave-70st krb5kdc[21689](info): commencing operation
krb5kdc: starting...
W20260924 05:03:01.125536 392 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790225929848825-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790225929848825-392-0/krb5kdc/test-user.keytab.
Sep 24 05:03:01 dist-test-slave-70st krb5kdc[21689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226181, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:03:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:03:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790225929848825-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790225929848825-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 05:03:01.235980 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790225929848825-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790225929848825-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790225929848825-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44809
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790225929848825-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44809
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:03:01.312513 21702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:03:01.312698 21702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:03:01.312719 21702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:03:01.313817 21702 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:03:01.313855 21702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:03:01.313868 21702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:03:01.313881 21702 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:03:01.313903 21702 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:03:01.315317 21702 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790225929848825-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790225929848825-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44809
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790225929848825-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44809
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21702
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790225929848825-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:03:01.315522 21702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:03:01.315711 21702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:03:01.320106 21708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:03:01.320138 21707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:03:01.320112 21710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:03:01.320549 21702 server_base.cc:1034] running on GCE node
I20260924 05:03:01.320772 21702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:03:01.321022 21702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:03:01.322196 21702 hybrid_clock.cc:648] HybridClock initialized: now 1790226181322177 us; error 28 us; skew 500 ppm
Sep 24 05:03:01 dist-test-slave-70st krb5kdc[21689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226181, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:03:01.331699 21702 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 05:03:01.332108 21702 webserver.cc:460] Webserver started at http://127.0.0.1:42527/ using document root <none> and password file <none>
I20260924 05:03:01.332309 21702 fs_manager.cc:359] Metadata directory not provided
I20260924 05:03:01.332350 21702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:03:01.332440 21702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:03:01.333465 21702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790225929848825-392-0/minicluster-data/master-0/data/instance:
uuid: "81753e9cd4fd4312834baec1d577242c"
format_stamp: "Formatted at 2026-09-24 05:03:01 on dist-test-slave-70st"
server_key: "f18bfbe6e20c69aad9e5d6aaa3d018e0"
server_key_iv: "a4b56c72958b665e93c3ed6fb4592e96"
server_key_version: "encryptionkey@0"
I20260924 05:03:01.333755 21702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790225929848825-392-0/minicluster-data/master-0/wal/instance:
uuid: "81753e9cd4fd4312834baec1d577242c"
format_stamp: "Formatted at 2026-09-24 05:03:01 on dist-test-slave-70st"
server_key: "f18bfbe6e20c69aad9e5d6aaa3d018e0"
server_key_iv: "a4b56c72958b665e93c3ed6fb4592e96"
server_key_version: "encryptionkey@0"
I20260924 05:03:01.334892 21702 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 05:03:01.335562 21717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:03:01.335712 21702 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:03:01.335785 21702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790225929848825-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790225929848825-392-0/minicluster-data/master-0/wal
uuid: "81753e9cd4fd4312834baec1d577242c"
format_stamp: "Formatted at 2026-09-24 05:03:01 on dist-test-slave-70st"
server_key: "f18bfbe6e20c69aad9e5d6aaa3d018e0"
server_key_iv: "a4b56c72958b665e93c3ed6fb4592e96"
server_key_version: "encryptionkey@0"
I20260924 05:03:01.335840 21702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790225929848825-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790225929848825-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790225929848825-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:03:01.348275 21702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:03:01.349937 21702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:03:01.350040 21702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:03:01.353492 21702 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44809
I20260924 05:03:01.353535 21769 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44809 every 8 connection(s)
I20260924 05:03:01.353832 21702 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:03:01.354784 21770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:03:01.356932 21770 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81753e9cd4fd4312834baec1d577242c: Bootstrap starting.
I20260924 05:03:01.357491 21770 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 81753e9cd4fd4312834baec1d577242c: Neither blocks nor log segments found. Creating new log.
I20260924 05:03:01.357736 21770 log.cc:824] T 00000000000000000000000000000000 P 81753e9cd4fd4312834baec1d577242c: Log is configured to *not* fsync() on all Append() calls
I20260924 05:03:01.358371 21770 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81753e9cd4fd4312834baec1d577242c: No bootstrap required, opened a new log
I20260924 05:03:01.359838 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 21702
I20260924 05:03:01.359942 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790225929848825-392-0/minicluster-data/master-0/wal/instance
I20260924 05:03:01.359938 21770 raft_consensus.cc:348] T 00000000000000000000000000000000 P 81753e9cd4fd4312834baec1d577242c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81753e9cd4fd4312834baec1d577242c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44809 } }
I20260924 05:03:01.360098 392 external_mini_cluster.cc:1428] Setting key dba1d1ccc8264380f3cffc8089fa32ca
I20260924 05:03:01.360087 21770 raft_consensus.cc:374] T 00000000000000000000000000000000 P 81753e9cd4fd4312834baec1d577242c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:03:01.360126 21770 raft_consensus.cc:729] T 00000000000000000000000000000000 P 81753e9cd4fd4312834baec1d577242c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81753e9cd4fd4312834baec1d577242c, State: Initialized, Role: FOLLOWER
I20260924 05:03:01.360263 21770 consensus_queue.cc:260] T 00000000000000000000000000000000 P 81753e9cd4fd4312834baec1d577242c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81753e9cd4fd4312834baec1d577242c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44809 } }
I20260924 05:03:01.360354 21770 raft_consensus.cc:388] T 00000000000000000000000000000000 P 81753e9cd4fd4312834baec1d577242c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:03:01.360383 21770 raft_consensus.cc:482] T 00000000000000000000000000000000 P 81753e9cd4fd4312834baec1d577242c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:03:01.360476 21770 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 81753e9cd4fd4312834baec1d577242c [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:03:01.361060 21770 raft_consensus.cc:504] T 00000000000000000000000000000000 P 81753e9cd4fd4312834baec1d577242c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81753e9cd4fd4312834baec1d577242c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44809 } }
I20260924 05:03:01.361157 21770 leader_election.cc:302] T 00000000000000000000000000000000 P 81753e9cd4fd4312834baec1d577242c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 81753e9cd4fd4312834baec1d577242c; no voters:
I20260924 05:03:01.361320 21770 leader_election.cc:288] T 00000000000000000000000000000000 P 81753e9cd4fd4312834baec1d577242c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:03:01.361340 21775 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 81753e9cd4fd4312834baec1d577242c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:03:01.361460 21775 raft_consensus.cc:686] T 00000000000000000000000000000000 P 81753e9cd4fd4312834baec1d577242c [term 1 LEADER]: Becoming Leader. State: Replica: 81753e9cd4fd4312834baec1d577242c, State: Running, Role: LEADER
I20260924 05:03:01.361541 21775 consensus_queue.cc:237] T 00000000000000000000000000000000 P 81753e9cd4fd4312834baec1d577242c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81753e9cd4fd4312834baec1d577242c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44809 } }
I20260924 05:03:01.361579 21770 sys_catalog.cc:564] T 00000000000000000000000000000000 P 81753e9cd4fd4312834baec1d577242c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:03:01.361888 21775 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81753e9cd4fd4312834baec1d577242c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "81753e9cd4fd4312834baec1d577242c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81753e9cd4fd4312834baec1d577242c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44809 } } }
I20260924 05:03:01.361908 21776 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81753e9cd4fd4312834baec1d577242c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81753e9cd4fd4312834baec1d577242c. Latest consensus state: current_term: 1 leader_uuid: "81753e9cd4fd4312834baec1d577242c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81753e9cd4fd4312834baec1d577242c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44809 } } }
I20260924 05:03:01.362030 21776 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81753e9cd4fd4312834baec1d577242c [sys.catalog]: This master's current role is: LEADER
I20260924 05:03:01.362219 21775 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81753e9cd4fd4312834baec1d577242c [sys.catalog]: This master's current role is: LEADER
I20260924 05:03:01.362375 21780 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:03:01.362910 21780 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:03:01.364370 21780 catalog_manager.cc:1347] Generated new cluster ID: e7afa3e7d9cb46f793e90faa202dffff
I20260924 05:03:01.364416 21780 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:03:01.365726 21774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:01.360522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.192:56635 (local address 127.0.0.1:44809)
0924 05:03:01.360702 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:01.360712 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:01.360737 (+ 25us) server_negotiation.cc:407] Connection header received
0924 05:03:01.361035 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:01.361040 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:01.361084 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:01.361212 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:03:01.361496 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:01.362835 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:01.363928 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:01.364371 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:01.364472 (+ 101us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:01.365107 (+ 635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:03:01.365111 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:03:01.365112 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:03:01.365269 (+ 157us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:03:01.365308 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:03:01.365318 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:03:01.365384 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 05:03:01.365527 (+ 143us) server_negotiation.cc:299] Negotiation successful
0924 05:03:01.365556 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":92,"threads_started":1}
I20260924 05:03:01.366650 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 05:03:01.372047 21774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:01.367871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.192:34187 (local address 127.0.0.1:44809)
0924 05:03:01.367903 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:01.367905 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:01.369584 (+ 1679us) server_negotiation.cc:407] Connection header received
0924 05:03:01.369638 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:01.369639 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:01.369649 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:01.369692 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:03:01.369815 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:01.370285 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:01.370959 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:01.371180 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:01.371231 (+ 51us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:01.371830 (+ 599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:03:01.371831 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:03:01.371831 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:03:01.371835 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:03:01.371848 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:03:01.371852 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:03:01.371900 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 05:03:01.371966 (+ 66us) server_negotiation.cc:299] Negotiation successful
0924 05:03:01.371976 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 05:03:01.375563 21780 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:03:01.376233 21780 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:03:01.381722 21780 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 81753e9cd4fd4312834baec1d577242c: Generated new TSK 0
I20260924 05:03:01.381889 21780 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 05:03:01.383699 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 21702
W20260924 05:03:01.387413 21687 connection.cc:535] client connection to 127.0.0.1:44809 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44809): Connection reset by peer (error 104)
2026-09-24T05:03:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1201 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790225929848825-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:03:01 dist-test-slave-70st krb5kdc[21801](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:03:01 dist-test-slave-70st krb5kdc[21801](info): set up 2 sockets
Sep 24 05:03:01 dist-test-slave-70st krb5kdc[21801](info): commencing operation
krb5kdc: starting...
W20260924 05:03:02.329648 392 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790225929848825-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790225929848825-392-0/krb5kdc/test-user.keytab.
Sep 24 05:03:02 dist-test-slave-70st krb5kdc[21801](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226182, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:03:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:03:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790225929848825-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790225929848825-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 05:03:02.440025 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790225929848825-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790225929848825-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790225929848825-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43621
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790225929848825-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43621
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:03:02.517544 21814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:03:02.517746 21814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:03:02.517783 21814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:03:02.518960 21814 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:03:02.519003 21814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:03:02.519022 21814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:03:02.519037 21814 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:03:02.519052 21814 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:03:02.520485 21814 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790225929848825-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790225929848825-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43621
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790225929848825-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43621
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21814
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790225929848825-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:03:02.520715 21814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:03:02.520916 21814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:03:02.525408 21820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:03:02.525352 21822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:03:02.525343 21819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:03:02.525624 21814 server_base.cc:1034] running on GCE node
I20260924 05:03:02.525869 21814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:03:02.526129 21814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:03:02.527307 21814 hybrid_clock.cc:648] HybridClock initialized: now 1790226182527284 us; error 32 us; skew 500 ppm
Sep 24 05:03:02 dist-test-slave-70st krb5kdc[21801](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226182, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:03:02.529590 21814 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 05:03:02.529976 21814 webserver.cc:460] Webserver started at http://127.0.0.1:45551/ using document root <none> and password file <none>
I20260924 05:03:02.530150 21814 fs_manager.cc:359] Metadata directory not provided
I20260924 05:03:02.530187 21814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:03:02.530273 21814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:03:02.531195 21814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790225929848825-392-0/minicluster-data/master-0/data/instance:
uuid: "8ec629eb07d04227b6464a2786eae97f"
format_stamp: "Formatted at 2026-09-24 05:03:02 on dist-test-slave-70st"
server_key: "9f6ef548b1899b67a2f288dc4b227629"
server_key_iv: "e52eff8535d6b77ed428a95c6ce044dd"
server_key_version: "encryptionkey@0"
I20260924 05:03:02.531472 21814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790225929848825-392-0/minicluster-data/master-0/wal/instance:
uuid: "8ec629eb07d04227b6464a2786eae97f"
format_stamp: "Formatted at 2026-09-24 05:03:02 on dist-test-slave-70st"
server_key: "9f6ef548b1899b67a2f288dc4b227629"
server_key_iv: "e52eff8535d6b77ed428a95c6ce044dd"
server_key_version: "encryptionkey@0"
I20260924 05:03:02.532599 21814 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 05:03:02.533229 21829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:03:02.533375 21814 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 05:03:02.533473 21814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790225929848825-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790225929848825-392-0/minicluster-data/master-0/wal
uuid: "8ec629eb07d04227b6464a2786eae97f"
format_stamp: "Formatted at 2026-09-24 05:03:02 on dist-test-slave-70st"
server_key: "9f6ef548b1899b67a2f288dc4b227629"
server_key_iv: "e52eff8535d6b77ed428a95c6ce044dd"
server_key_version: "encryptionkey@0"
I20260924 05:03:02.533532 21814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790225929848825-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790225929848825-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790225929848825-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:03:02.536571 21814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:03:02.538384 21814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:03:02.538501 21814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:03:02.542110 21814 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43621
I20260924 05:03:02.542153 21881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43621 every 8 connection(s)
I20260924 05:03:02.542420 21814 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:03:02.543272 21882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:03:02.544224 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 21814
I20260924 05:03:02.544359 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790225929848825-392-0/minicluster-data/master-0/wal/instance
I20260924 05:03:02.544535 392 external_mini_cluster.cc:1428] Setting key b544df629ba3b14d88d8a2f661085c03
I20260924 05:03:02.545851 21882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ec629eb07d04227b6464a2786eae97f: Bootstrap starting.
I20260924 05:03:02.546473 21882 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ec629eb07d04227b6464a2786eae97f: Neither blocks nor log segments found. Creating new log.
I20260924 05:03:02.546752 21882 log.cc:824] T 00000000000000000000000000000000 P 8ec629eb07d04227b6464a2786eae97f: Log is configured to *not* fsync() on all Append() calls
I20260924 05:03:02.546943 21885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:02.544999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.192:59845 (local address 127.0.0.1:43621)
0924 05:03:02.545143 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:02.545150 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:02.545174 (+ 24us) server_negotiation.cc:407] Connection header received
0924 05:03:02.545643 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:02.545649 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:02.545675 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:02.545790 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:03:02.546374 (+ 584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:03:02.546377 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:03:02.546378 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:03:02.546381 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:03:02.546416 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:03:02.546426 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:03:02.546487 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 05:03:02.546742 (+ 255us) server_negotiation.cc:299] Negotiation successful
0924 05:03:02.546786 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":60,"threads_started":1}
I20260924 05:03:02.547356 21882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ec629eb07d04227b6464a2786eae97f: No bootstrap required, opened a new log
I20260924 05:03:02.547822 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 05:03:02.549322 21882 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ec629eb07d04227b6464a2786eae97f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ec629eb07d04227b6464a2786eae97f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43621 } }
I20260924 05:03:02.549556 21882 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8ec629eb07d04227b6464a2786eae97f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:03:02.549636 21882 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ec629eb07d04227b6464a2786eae97f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ec629eb07d04227b6464a2786eae97f, State: Initialized, Role: FOLLOWER
I20260924 05:03:02.549777 21882 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ec629eb07d04227b6464a2786eae97f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ec629eb07d04227b6464a2786eae97f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43621 } }
I20260924 05:03:02.549871 21882 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8ec629eb07d04227b6464a2786eae97f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:03:02.549909 21882 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8ec629eb07d04227b6464a2786eae97f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:03:02.549960 21882 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ec629eb07d04227b6464a2786eae97f [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:03:02.550094 21885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:02.549057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.192:44425 (local address 127.0.0.1:43621)
0924 05:03:02.549089 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:02.549090 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:02.549199 (+ 109us) server_negotiation.cc:407] Connection header received
0924 05:03:02.549237 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:02.549238 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:02.549242 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:02.549308 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:03:02.549862 (+ 554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:03:02.549863 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:03:02.549864 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:03:02.549867 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:03:02.549881 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:03:02.549884 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:03:02.549938 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 05:03:02.550021 (+ 83us) server_negotiation.cc:299] Negotiation successful
0924 05:03:02.550035 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260924 05:03:02.550558 21882 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8ec629eb07d04227b6464a2786eae97f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ec629eb07d04227b6464a2786eae97f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43621 } }
I20260924 05:03:02.550640 21882 leader_election.cc:302] T 00000000000000000000000000000000 P 8ec629eb07d04227b6464a2786eae97f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8ec629eb07d04227b6464a2786eae97f; no voters:
I20260924 05:03:02.550791 21882 leader_election.cc:288] T 00000000000000000000000000000000 P 8ec629eb07d04227b6464a2786eae97f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:03:02.550822 21892 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8ec629eb07d04227b6464a2786eae97f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:03:02.550941 21892 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8ec629eb07d04227b6464a2786eae97f [term 1 LEADER]: Becoming Leader. State: Replica: 8ec629eb07d04227b6464a2786eae97f, State: Running, Role: LEADER
I20260924 05:03:02.551003 21882 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ec629eb07d04227b6464a2786eae97f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:03:02.551046 21892 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ec629eb07d04227b6464a2786eae97f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ec629eb07d04227b6464a2786eae97f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43621 } }
I20260924 05:03:02.551378 21894 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ec629eb07d04227b6464a2786eae97f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ec629eb07d04227b6464a2786eae97f. Latest consensus state: current_term: 1 leader_uuid: "8ec629eb07d04227b6464a2786eae97f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ec629eb07d04227b6464a2786eae97f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43621 } } }
I20260924 05:03:02.551506 21894 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ec629eb07d04227b6464a2786eae97f [sys.catalog]: This master's current role is: LEADER
I20260924 05:03:02.551375 21893 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ec629eb07d04227b6464a2786eae97f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8ec629eb07d04227b6464a2786eae97f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ec629eb07d04227b6464a2786eae97f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43621 } } }
I20260924 05:03:02.551621 21893 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ec629eb07d04227b6464a2786eae97f [sys.catalog]: This master's current role is: LEADER
I20260924 05:03:02.551941 21902 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:03:02.552392 21902 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:03:02.553782 21902 catalog_manager.cc:1347] Generated new cluster ID: 5fcfa44b966c4aada2ce04228090b31b
I20260924 05:03:02.553838 21902 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:03:02.577180 21902 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:03:02.577369 21902 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:03:02.587107 21902 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8ec629eb07d04227b6464a2786eae97f: Generated new TSK 0
I20260924 05:03:02.587309 21902 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 05:03:02.594020 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 21814
2026-09-24T05:03:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1210 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790225929848825-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:03:02 dist-test-slave-70st krb5kdc[21913](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:03:02 dist-test-slave-70st krb5kdc[21913](info): set up 2 sockets
Sep 24 05:03:02 dist-test-slave-70st krb5kdc[21913](info): commencing operation
krb5kdc: starting...
W20260924 05:03:03.537400 392 mini_kdc.cc:121] Time spent starting KDC: real 0.926s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790225929848825-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790225929848825-392-0/krb5kdc/test-user.keytab.
Sep 24 05:03:03 dist-test-slave-70st krb5kdc[21913](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226183, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:03:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:03:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 05:03:03.648204 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790225929848825-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790225929848825-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790225929848825-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42015
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42015
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:03:03.726994 21926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:03:03.727272 21926 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:03:03.727313 21926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:03:03.728616 21926 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:03:03.728663 21926 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:03:03.728681 21926 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:03:03.728698 21926 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:03:03.728713 21926 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:03:03.730221 21926 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790225929848825-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790225929848825-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42015
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42015
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21926
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790225929848825-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:03:03.730473 21926 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:03:03.730685 21926 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:03:03.735378 21932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:03:03.735378 21934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:03:03.735436 21926 server_base.cc:1034] running on GCE node
W20260924 05:03:03.735380 21931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:03:03.735811 21926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:03:03.736068 21926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:03:03.737211 21926 hybrid_clock.cc:648] HybridClock initialized: now 1790226183737198 us; error 31 us; skew 500 ppm
Sep 24 05:03:03 dist-test-slave-70st krb5kdc[21913](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226183, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:03:03.739568 21926 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 05:03:03.739977 21926 webserver.cc:460] Webserver started at http://127.0.98.62:32963/ using document root <none> and password file <none>
I20260924 05:03:03.740166 21926 fs_manager.cc:359] Metadata directory not provided
I20260924 05:03:03.740206 21926 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:03:03.740294 21926 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:03:03.741323 21926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790225929848825-392-0/minicluster-data/master-0/data/instance:
uuid: "6c036fa2d8bc481c86bbbe689c964ad1"
format_stamp: "Formatted at 2026-09-24 05:03:03 on dist-test-slave-70st"
server_key: "24839769f33816038796d803a7551856"
server_key_iv: "e068d6f0113cf83dcfe8594ccb48451c"
server_key_version: "encryptionkey@0"
I20260924 05:03:03.741659 21926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790225929848825-392-0/minicluster-data/master-0/wal/instance:
uuid: "6c036fa2d8bc481c86bbbe689c964ad1"
format_stamp: "Formatted at 2026-09-24 05:03:03 on dist-test-slave-70st"
server_key: "24839769f33816038796d803a7551856"
server_key_iv: "e068d6f0113cf83dcfe8594ccb48451c"
server_key_version: "encryptionkey@0"
I20260924 05:03:03.742842 21926 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 05:03:03.743561 21941 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:03:03.743734 21926 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:03:03.743821 21926 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790225929848825-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790225929848825-392-0/minicluster-data/master-0/wal
uuid: "6c036fa2d8bc481c86bbbe689c964ad1"
format_stamp: "Formatted at 2026-09-24 05:03:03 on dist-test-slave-70st"
server_key: "24839769f33816038796d803a7551856"
server_key_iv: "e068d6f0113cf83dcfe8594ccb48451c"
server_key_version: "encryptionkey@0"
I20260924 05:03:03.743880 21926 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790225929848825-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790225929848825-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790225929848825-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:03:03.758795 21926 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:03:03.760627 21926 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:03:03.760735 21926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:03:03.764652 21926 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42015
I20260924 05:03:03.764734 21993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42015 every 8 connection(s)
I20260924 05:03:03.765034 21926 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:03:03.765926 21994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:03:03.768074 21994 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c036fa2d8bc481c86bbbe689c964ad1: Bootstrap starting.
I20260924 05:03:03.768621 21994 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c036fa2d8bc481c86bbbe689c964ad1: Neither blocks nor log segments found. Creating new log.
I20260924 05:03:03.768867 21994 log.cc:824] T 00000000000000000000000000000000 P 6c036fa2d8bc481c86bbbe689c964ad1: Log is configured to *not* fsync() on all Append() calls
I20260924 05:03:03.769603 21994 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c036fa2d8bc481c86bbbe689c964ad1: No bootstrap required, opened a new log
I20260924 05:03:03.771126 21994 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c036fa2d8bc481c86bbbe689c964ad1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c036fa2d8bc481c86bbbe689c964ad1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42015 } }
I20260924 05:03:03.771265 21994 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6c036fa2d8bc481c86bbbe689c964ad1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:03:03.771286 21994 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c036fa2d8bc481c86bbbe689c964ad1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c036fa2d8bc481c86bbbe689c964ad1, State: Initialized, Role: FOLLOWER
I20260924 05:03:03.771425 21994 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c036fa2d8bc481c86bbbe689c964ad1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c036fa2d8bc481c86bbbe689c964ad1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42015 } }
I20260924 05:03:03.771485 21994 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6c036fa2d8bc481c86bbbe689c964ad1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:03:03.771507 21994 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c036fa2d8bc481c86bbbe689c964ad1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:03:03.771551 21994 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c036fa2d8bc481c86bbbe689c964ad1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:03:03.772040 21994 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c036fa2d8bc481c86bbbe689c964ad1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c036fa2d8bc481c86bbbe689c964ad1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42015 } }
I20260924 05:03:03.772126 21994 leader_election.cc:302] T 00000000000000000000000000000000 P 6c036fa2d8bc481c86bbbe689c964ad1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c036fa2d8bc481c86bbbe689c964ad1; no voters:
I20260924 05:03:03.772284 21994 leader_election.cc:288] T 00000000000000000000000000000000 P 6c036fa2d8bc481c86bbbe689c964ad1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:03:03.772323 21997 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c036fa2d8bc481c86bbbe689c964ad1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:03:03.772491 21997 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c036fa2d8bc481c86bbbe689c964ad1 [term 1 LEADER]: Becoming Leader. State: Replica: 6c036fa2d8bc481c86bbbe689c964ad1, State: Running, Role: LEADER
I20260924 05:03:03.772537 21994 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c036fa2d8bc481c86bbbe689c964ad1 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:03:03.772585 21997 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c036fa2d8bc481c86bbbe689c964ad1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c036fa2d8bc481c86bbbe689c964ad1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42015 } }
I20260924 05:03:03.772809 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 21926
I20260924 05:03:03.772936 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790225929848825-392-0/minicluster-data/master-0/wal/instance
I20260924 05:03:03.773059 21999 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c036fa2d8bc481c86bbbe689c964ad1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c036fa2d8bc481c86bbbe689c964ad1. Latest consensus state: current_term: 1 leader_uuid: "6c036fa2d8bc481c86bbbe689c964ad1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c036fa2d8bc481c86bbbe689c964ad1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42015 } } }
I20260924 05:03:03.773146 21999 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c036fa2d8bc481c86bbbe689c964ad1 [sys.catalog]: This master's current role is: LEADER
I20260924 05:03:03.773165 392 external_mini_cluster.cc:1428] Setting key 0ea9bd43d9123c29adbcf2298d7f327c
I20260924 05:03:03.773164 21998 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c036fa2d8bc481c86bbbe689c964ad1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6c036fa2d8bc481c86bbbe689c964ad1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c036fa2d8bc481c86bbbe689c964ad1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42015 } } }
I20260924 05:03:03.773209 21998 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c036fa2d8bc481c86bbbe689c964ad1 [sys.catalog]: This master's current role is: LEADER
I20260924 05:03:03.773358 22001 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:03:03.773957 22001 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:03:03.775595 22001 catalog_manager.cc:1347] Generated new cluster ID: 1a240a6ac7fe4712a1028419b6d11795
I20260924 05:03:03.775653 22001 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 05:03:03 dist-test-slave-70st krb5kdc[21913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226183, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:03:03.783267 22013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:03.773930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56265 (local address 127.0.98.62:42015)
0924 05:03:03.774442 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:03.774451 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:03.774484 (+ 33us) server_negotiation.cc:407] Connection header received
0924 05:03:03.774599 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:03.774615 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:03.774676 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:03.774827 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:03:03.775616 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:03.776522 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:03.777240 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:03.777726 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:03.777831 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:03.779931 (+ 2100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:03:03.779935 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:03:03.779935 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:03:03.779946 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:03:03.781471 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:03.781814 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:03.781817 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:03.781818 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:03.782284 (+ 466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:03.782498 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:03.782500 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:03.782500 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:03.782729 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:03:03.782898 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0924 05:03:03.783019 (+ 121us) server_negotiation.cc:299] Negotiation successful
0924 05:03:03.783115 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":77,"threads_started":1}
I20260924 05:03:03.784353 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 05:03:03.794376 22013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:03.785648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44735 (local address 127.0.98.62:42015)
0924 05:03:03.785694 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:03.785696 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:03.788915 (+ 3219us) server_negotiation.cc:407] Connection header received
0924 05:03:03.788979 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:03.788980 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:03.788990 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:03.789037 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:03:03.789573 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:03.790086 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:03.790764 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:03.790968 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:03.791022 (+ 54us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:03.792107 (+ 1085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:03:03.792108 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:03:03.792109 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:03:03.792116 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:03:03.793347 (+ 1231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:03.793691 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:03.793692 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:03.793692 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:03.793742 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:03.793922 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:03.793922 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:03.793923 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:03.794023 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:03:03.794122 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 05:03:03.794224 (+ 102us) server_negotiation.cc:299] Negotiation successful
0924 05:03:03.794284 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 05:03:03.803387 22001 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:03:03.803913 22001 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:03:03.808553 22001 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6c036fa2d8bc481c86bbbe689c964ad1: Generated new TSK 0
I20260924 05:03:03.808748 22001 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 05:03:03.814922 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 21926
W20260924 05:03:03.818553 21911 connection.cc:535] client connection to 127.0.98.62:42015 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:42015): Connection reset by peer (error 104)
2026-09-24T05:03:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1220 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790225929848825-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:03:03 dist-test-slave-70st krb5kdc[22025](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:03:03 dist-test-slave-70st krb5kdc[22025](info): set up 2 sockets
Sep 24 05:03:03 dist-test-slave-70st krb5kdc[22025](info): commencing operation
krb5kdc: starting...
W20260924 05:03:04.761808 392 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790225929848825-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790225929848825-392-0/krb5kdc/test-user.keytab.
Sep 24 05:03:04 dist-test-slave-70st krb5kdc[22025](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226184, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:03:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:03:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 05:03:04.872092 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790225929848825-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790225929848825-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790225929848825-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39839
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39839
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:03:04.948944 22038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:03:04.949124 22038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:03:04.949146 22038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:03:04.950331 22038 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:03:04.950373 22038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:03:04.950387 22038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:03:04.950397 22038 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:03:04.950419 22038 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:03:04.951833 22038 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790225929848825-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790225929848825-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39839
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22038
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790225929848825-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:03:04.952039 22038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:03:04.952227 22038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:03:04.956705 22046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:03:04.956789 22038 server_base.cc:1034] running on GCE node
W20260924 05:03:04.956806 22044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:03:04.956707 22043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:03:04.957124 22038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:03:04.957404 22038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:03:04.958559 22038 hybrid_clock.cc:648] HybridClock initialized: now 1790226184958550 us; error 27 us; skew 500 ppm
Sep 24 05:03:04 dist-test-slave-70st krb5kdc[22025](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226184, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:03:04.960783 22038 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 05:03:04.961203 22038 webserver.cc:460] Webserver started at http://127.0.98.62:41419/ using document root <none> and password file <none>
I20260924 05:03:04.961447 22038 fs_manager.cc:359] Metadata directory not provided
I20260924 05:03:04.961503 22038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:03:04.961620 22038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:03:04.962580 22038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790225929848825-392-0/minicluster-data/master-0/data/instance:
uuid: "86b6619491664d8c965ee04382002763"
format_stamp: "Formatted at 2026-09-24 05:03:04 on dist-test-slave-70st"
server_key: "f8a5ad2f92a230a173f30d0b689c8def"
server_key_iv: "8216455357f6d90ab256de6e76c58fb3"
server_key_version: "encryptionkey@0"
I20260924 05:03:04.962888 22038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790225929848825-392-0/minicluster-data/master-0/wal/instance:
uuid: "86b6619491664d8c965ee04382002763"
format_stamp: "Formatted at 2026-09-24 05:03:04 on dist-test-slave-70st"
server_key: "f8a5ad2f92a230a173f30d0b689c8def"
server_key_iv: "8216455357f6d90ab256de6e76c58fb3"
server_key_version: "encryptionkey@0"
I20260924 05:03:04.964041 22038 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260924 05:03:04.964685 22053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:03:04.964824 22038 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 05:03:04.964901 22038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790225929848825-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790225929848825-392-0/minicluster-data/master-0/wal
uuid: "86b6619491664d8c965ee04382002763"
format_stamp: "Formatted at 2026-09-24 05:03:04 on dist-test-slave-70st"
server_key: "f8a5ad2f92a230a173f30d0b689c8def"
server_key_iv: "8216455357f6d90ab256de6e76c58fb3"
server_key_version: "encryptionkey@0"
I20260924 05:03:04.964964 22038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790225929848825-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790225929848825-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790225929848825-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:03:04.980022 22038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:03:04.981794 22038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:03:04.981925 22038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:03:04.985472 22038 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39839
I20260924 05:03:04.985517 22105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39839 every 8 connection(s)
I20260924 05:03:04.985836 22038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:03:04.986032 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 22038
I20260924 05:03:04.986140 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790225929848825-392-0/minicluster-data/master-0/wal/instance
I20260924 05:03:04.986330 392 external_mini_cluster.cc:1428] Setting key d28f8705b8881a8b59d9272142b6a7c5
I20260924 05:03:04.986742 22106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:03:04.989852 22106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86b6619491664d8c965ee04382002763: Bootstrap starting.
I20260924 05:03:04.990530 22106 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 86b6619491664d8c965ee04382002763: Neither blocks nor log segments found. Creating new log.
I20260924 05:03:04.990825 22106 log.cc:824] T 00000000000000000000000000000000 P 86b6619491664d8c965ee04382002763: Log is configured to *not* fsync() on all Append() calls
I20260924 05:03:04.991591 22106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86b6619491664d8c965ee04382002763: No bootstrap required, opened a new log
I20260924 05:03:04.991961 22109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:04.986783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57413 (local address 127.0.98.62:39839)
0924 05:03:04.986967 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:04.986976 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:04.987006 (+ 30us) server_negotiation.cc:407] Connection header received
0924 05:03:04.987511 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:04.987520 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:04.987583 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:04.987730 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:03:04.987932 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:04.989230 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:04.990045 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:04.990626 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:04.990745 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:04.991409 (+ 664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:03:04.991413 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:03:04.991414 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:03:04.991565 (+ 151us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:03:04.991595 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:03:04.991604 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:03:04.991671 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 05:03:04.991776 (+ 105us) server_negotiation.cc:299] Negotiation successful
0924 05:03:04.991802 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":87,"threads_started":1}
I20260924 05:03:04.993038 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 05:03:04.993184 22106 raft_consensus.cc:348] T 00000000000000000000000000000000 P 86b6619491664d8c965ee04382002763 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86b6619491664d8c965ee04382002763" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39839 } }
I20260924 05:03:04.993316 22106 raft_consensus.cc:374] T 00000000000000000000000000000000 P 86b6619491664d8c965ee04382002763 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:03:04.993350 22106 raft_consensus.cc:729] T 00000000000000000000000000000000 P 86b6619491664d8c965ee04382002763 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86b6619491664d8c965ee04382002763, State: Initialized, Role: FOLLOWER
I20260924 05:03:04.993484 22106 consensus_queue.cc:260] T 00000000000000000000000000000000 P 86b6619491664d8c965ee04382002763 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86b6619491664d8c965ee04382002763" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39839 } }
I20260924 05:03:04.993556 22106 raft_consensus.cc:388] T 00000000000000000000000000000000 P 86b6619491664d8c965ee04382002763 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:03:04.993587 22106 raft_consensus.cc:482] T 00000000000000000000000000000000 P 86b6619491664d8c965ee04382002763 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:03:04.993633 22106 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 86b6619491664d8c965ee04382002763 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:03:04.994166 22106 raft_consensus.cc:504] T 00000000000000000000000000000000 P 86b6619491664d8c965ee04382002763 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86b6619491664d8c965ee04382002763" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39839 } }
I20260924 05:03:04.994272 22106 leader_election.cc:302] T 00000000000000000000000000000000 P 86b6619491664d8c965ee04382002763 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 86b6619491664d8c965ee04382002763; no voters:
I20260924 05:03:04.994446 22106 leader_election.cc:288] T 00000000000000000000000000000000 P 86b6619491664d8c965ee04382002763 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:03:04.994479 22116 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 86b6619491664d8c965ee04382002763 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:03:04.994634 22116 raft_consensus.cc:686] T 00000000000000000000000000000000 P 86b6619491664d8c965ee04382002763 [term 1 LEADER]: Becoming Leader. State: Replica: 86b6619491664d8c965ee04382002763, State: Running, Role: LEADER
I20260924 05:03:04.994760 22106 sys_catalog.cc:564] T 00000000000000000000000000000000 P 86b6619491664d8c965ee04382002763 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:03:04.994762 22116 consensus_queue.cc:237] T 00000000000000000000000000000000 P 86b6619491664d8c965ee04382002763 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86b6619491664d8c965ee04382002763" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39839 } }
I20260924 05:03:04.995447 22117 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86b6619491664d8c965ee04382002763 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "86b6619491664d8c965ee04382002763" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86b6619491664d8c965ee04382002763" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39839 } } }
I20260924 05:03:04.995472 22118 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86b6619491664d8c965ee04382002763 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86b6619491664d8c965ee04382002763. Latest consensus state: current_term: 1 leader_uuid: "86b6619491664d8c965ee04382002763" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86b6619491664d8c965ee04382002763" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39839 } } }
I20260924 05:03:04.995631 22118 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86b6619491664d8c965ee04382002763 [sys.catalog]: This master's current role is: LEADER
I20260924 05:03:04.995570 22117 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86b6619491664d8c965ee04382002763 [sys.catalog]: This master's current role is: LEADER
I20260924 05:03:04.995850 22125 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:03:04.996287 22125 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:03:04.997658 22109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:04.994326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44365 (local address 127.0.98.62:39839)
0924 05:03:04.994361 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:04.994363 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:04.994483 (+ 120us) server_negotiation.cc:407] Connection header received
0924 05:03:04.994530 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:04.994531 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:04.994543 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:04.994586 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:03:04.994776 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:04.995343 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:04.996320 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:04.996594 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:04.996635 (+ 41us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:04.997367 (+ 732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:03:04.997368 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:03:04.997369 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:03:04.997374 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:03:04.997389 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:03:04.997393 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:03:04.997470 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 05:03:04.997575 (+ 105us) server_negotiation.cc:299] Negotiation successful
0924 05:03:04.997585 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 05:03:04.997920 22125 catalog_manager.cc:1347] Generated new cluster ID: d585406182bf493cb3df1f6df5f754b7
I20260924 05:03:04.998072 22125 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:03:05.012406 22125 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:03:05.012918 22125 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:03:05.025064 22125 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 86b6619491664d8c965ee04382002763: Generated new TSK 0
I20260924 05:03:05.025281 22125 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 05:03:05.036108 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 22038
W20260924 05:03:05.039849 22023 connection.cc:535] client connection to 127.0.98.62:39839 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:39839): Connection reset by peer (error 104)
2026-09-24T05:03:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1221 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790225929848825-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:03:05 dist-test-slave-70st krb5kdc[22137](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:03:05 dist-test-slave-70st krb5kdc[22137](info): set up 2 sockets
Sep 24 05:03:05 dist-test-slave-70st krb5kdc[22137](info): commencing operation
krb5kdc: starting...
W20260924 05:03:05.981184 392 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790225929848825-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790225929848825-392-0/krb5kdc/test-user.keytab.
Sep 24 05:03:06 dist-test-slave-70st krb5kdc[22137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226186, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:03:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:03:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 05:03:06.095858 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790225929848825-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790225929848825-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790225929848825-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46775
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46775
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:03:06.173519 22150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:03:06.173717 22150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:03:06.173738 22150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:03:06.174866 22150 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:03:06.174909 22150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:03:06.174921 22150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:03:06.174932 22150 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:03:06.174944 22150 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:03:06.176384 22150 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790225929848825-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790225929848825-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46775
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22150
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790225929848825-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:03:06.176601 22150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:03:06.176798 22150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:03:06.181506 22158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:03:06.181632 22156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:03:06.181684 22150 server_base.cc:1034] running on GCE node
W20260924 05:03:06.181515 22155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:03:06.181989 22150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:03:06.182283 22150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:03:06.183447 22150 hybrid_clock.cc:648] HybridClock initialized: now 1790226186183429 us; error 30 us; skew 500 ppm
Sep 24 05:03:06 dist-test-slave-70st krb5kdc[22137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226186, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:03:06.185707 22150 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 05:03:06.186080 22150 webserver.cc:460] Webserver started at http://127.0.98.62:45681/ using document root <none> and password file <none>
I20260924 05:03:06.186257 22150 fs_manager.cc:359] Metadata directory not provided
I20260924 05:03:06.186295 22150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:03:06.186391 22150 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:03:06.187350 22150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790225929848825-392-0/minicluster-data/master-0/data/instance:
uuid: "351d2e1b12f1416a9a9db9f75982d19b"
format_stamp: "Formatted at 2026-09-24 05:03:06 on dist-test-slave-70st"
server_key: "e02b35db8c77dd9c1b55fd10a1b94a73"
server_key_iv: "fa05c4f5dc3539e2f6d3473fafd7e06b"
server_key_version: "encryptionkey@0"
I20260924 05:03:06.187629 22150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790225929848825-392-0/minicluster-data/master-0/wal/instance:
uuid: "351d2e1b12f1416a9a9db9f75982d19b"
format_stamp: "Formatted at 2026-09-24 05:03:06 on dist-test-slave-70st"
server_key: "e02b35db8c77dd9c1b55fd10a1b94a73"
server_key_iv: "fa05c4f5dc3539e2f6d3473fafd7e06b"
server_key_version: "encryptionkey@0"
I20260924 05:03:06.188745 22150 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 05:03:06.189527 22165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:03:06.189677 22150 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 05:03:06.189743 22150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790225929848825-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790225929848825-392-0/minicluster-data/master-0/wal
uuid: "351d2e1b12f1416a9a9db9f75982d19b"
format_stamp: "Formatted at 2026-09-24 05:03:06 on dist-test-slave-70st"
server_key: "e02b35db8c77dd9c1b55fd10a1b94a73"
server_key_iv: "fa05c4f5dc3539e2f6d3473fafd7e06b"
server_key_version: "encryptionkey@0"
I20260924 05:03:06.189812 22150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790225929848825-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790225929848825-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790225929848825-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:03:06.192858 22150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:03:06.194695 22150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:03:06.194806 22150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:03:06.198397 22150 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46775
I20260924 05:03:06.198423 22217 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46775 every 8 connection(s)
I20260924 05:03:06.198706 22150 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:03:06.199601 22218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:03:06.199618 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 22150
I20260924 05:03:06.199709 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790225929848825-392-0/minicluster-data/master-0/wal/instance
I20260924 05:03:06.199908 392 external_mini_cluster.cc:1428] Setting key ca011ff1a65df7b6317fd73a8b936059
I20260924 05:03:06.202127 22221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:06.200362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49431 (local address 127.0.98.62:46775)
0924 05:03:06.200537 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:06.200546 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:06.200575 (+ 29us) server_negotiation.cc:407] Connection header received
0924 05:03:06.200991 (+ 416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:06.200998 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:06.201023 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:06.201139 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:03:06.201748 (+ 609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:03:06.201752 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:03:06.201752 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:03:06.201756 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:03:06.201794 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:03:06.201804 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:03:06.201858 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 05:03:06.201937 (+ 79us) server_negotiation.cc:299] Negotiation successful
0924 05:03:06.201974 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":73,"threads_started":1}
I20260924 05:03:06.202620 22218 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 351d2e1b12f1416a9a9db9f75982d19b: Bootstrap starting.
I20260924 05:03:06.202984 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 05:03:06.203248 22218 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 351d2e1b12f1416a9a9db9f75982d19b: Neither blocks nor log segments found. Creating new log.
I20260924 05:03:06.203516 22218 log.cc:824] T 00000000000000000000000000000000 P 351d2e1b12f1416a9a9db9f75982d19b: Log is configured to *not* fsync() on all Append() calls
I20260924 05:03:06.204195 22218 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 351d2e1b12f1416a9a9db9f75982d19b: No bootstrap required, opened a new log
I20260924 05:03:06.205209 22221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:06.204264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37315 (local address 127.0.98.62:46775)
0924 05:03:06.204345 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:06.204347 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:06.204386 (+ 39us) server_negotiation.cc:407] Connection header received
0924 05:03:06.204453 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:06.204453 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:06.204457 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:06.204501 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:03:06.204985 (+ 484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:03:06.204986 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:03:06.204986 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:03:06.204990 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:03:06.205003 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:03:06.205006 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:03:06.205068 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 05:03:06.205133 (+ 65us) server_negotiation.cc:299] Negotiation successful
0924 05:03:06.205143 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 05:03:06.206256 22218 raft_consensus.cc:348] T 00000000000000000000000000000000 P 351d2e1b12f1416a9a9db9f75982d19b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "351d2e1b12f1416a9a9db9f75982d19b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46775 } }
I20260924 05:03:06.206395 22218 raft_consensus.cc:374] T 00000000000000000000000000000000 P 351d2e1b12f1416a9a9db9f75982d19b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:03:06.206415 22218 raft_consensus.cc:729] T 00000000000000000000000000000000 P 351d2e1b12f1416a9a9db9f75982d19b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 351d2e1b12f1416a9a9db9f75982d19b, State: Initialized, Role: FOLLOWER
I20260924 05:03:06.206491 22218 consensus_queue.cc:260] T 00000000000000000000000000000000 P 351d2e1b12f1416a9a9db9f75982d19b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "351d2e1b12f1416a9a9db9f75982d19b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46775 } }
I20260924 05:03:06.206545 22218 raft_consensus.cc:388] T 00000000000000000000000000000000 P 351d2e1b12f1416a9a9db9f75982d19b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:03:06.206569 22218 raft_consensus.cc:482] T 00000000000000000000000000000000 P 351d2e1b12f1416a9a9db9f75982d19b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:03:06.206599 22218 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 351d2e1b12f1416a9a9db9f75982d19b [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:03:06.207048 22218 raft_consensus.cc:504] T 00000000000000000000000000000000 P 351d2e1b12f1416a9a9db9f75982d19b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "351d2e1b12f1416a9a9db9f75982d19b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46775 } }
I20260924 05:03:06.207125 22218 leader_election.cc:302] T 00000000000000000000000000000000 P 351d2e1b12f1416a9a9db9f75982d19b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 351d2e1b12f1416a9a9db9f75982d19b; no voters:
I20260924 05:03:06.207247 22218 leader_election.cc:288] T 00000000000000000000000000000000 P 351d2e1b12f1416a9a9db9f75982d19b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:03:06.207289 22228 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 351d2e1b12f1416a9a9db9f75982d19b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:03:06.207438 22228 raft_consensus.cc:686] T 00000000000000000000000000000000 P 351d2e1b12f1416a9a9db9f75982d19b [term 1 LEADER]: Becoming Leader. State: Replica: 351d2e1b12f1416a9a9db9f75982d19b, State: Running, Role: LEADER
I20260924 05:03:06.207444 22218 sys_catalog.cc:564] T 00000000000000000000000000000000 P 351d2e1b12f1416a9a9db9f75982d19b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:03:06.207532 22228 consensus_queue.cc:237] T 00000000000000000000000000000000 P 351d2e1b12f1416a9a9db9f75982d19b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "351d2e1b12f1416a9a9db9f75982d19b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46775 } }
I20260924 05:03:06.207882 22229 sys_catalog.cc:455] T 00000000000000000000000000000000 P 351d2e1b12f1416a9a9db9f75982d19b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "351d2e1b12f1416a9a9db9f75982d19b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "351d2e1b12f1416a9a9db9f75982d19b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46775 } } }
I20260924 05:03:06.208107 22229 sys_catalog.cc:458] T 00000000000000000000000000000000 P 351d2e1b12f1416a9a9db9f75982d19b [sys.catalog]: This master's current role is: LEADER
I20260924 05:03:06.208045 22230 sys_catalog.cc:455] T 00000000000000000000000000000000 P 351d2e1b12f1416a9a9db9f75982d19b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 351d2e1b12f1416a9a9db9f75982d19b. Latest consensus state: current_term: 1 leader_uuid: "351d2e1b12f1416a9a9db9f75982d19b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "351d2e1b12f1416a9a9db9f75982d19b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46775 } } }
I20260924 05:03:06.208293 22230 sys_catalog.cc:458] T 00000000000000000000000000000000 P 351d2e1b12f1416a9a9db9f75982d19b [sys.catalog]: This master's current role is: LEADER
I20260924 05:03:06.208465 22237 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:03:06.208926 22237 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:03:06.210340 22237 catalog_manager.cc:1347] Generated new cluster ID: 72c2dc1511b44186a79ec320d8325559
I20260924 05:03:06.210390 22237 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:03:06.219758 22237 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:03:06.219913 22237 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:03:06.223325 22237 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 351d2e1b12f1416a9a9db9f75982d19b: Generated new TSK 0
I20260924 05:03:06.223472 22237 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 05:03:06.224735 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 22150
2026-09-24T05:03:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1188 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790225929848825-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:03:06 dist-test-slave-70st krb5kdc[22249](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:03:06 dist-test-slave-70st krb5kdc[22249](info): set up 2 sockets
Sep 24 05:03:06 dist-test-slave-70st krb5kdc[22249](info): commencing operation
krb5kdc: starting...
W20260924 05:03:07.169328 392 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790225929848825-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790225929848825-392-0/krb5kdc/test-user.keytab.
Sep 24 05:03:07 dist-test-slave-70st krb5kdc[22249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226187, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:03:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:03:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 05:03:07.277796 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790225929848825-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790225929848825-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790225929848825-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45519
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45519
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:03:07.355613 22262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:03:07.355809 22262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:03:07.355839 22262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:03:07.356966 22262 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:03:07.357007 22262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:03:07.357023 22262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:03:07.357038 22262 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:03:07.357054 22262 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:03:07.358530 22262 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790225929848825-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790225929848825-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45519
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22262
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790225929848825-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:03:07.358758 22262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:03:07.358946 22262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:03:07.363442 22267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:03:07.363473 22268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:03:07.363483 22270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:03:07.363528 22262 server_base.cc:1034] running on GCE node
I20260924 05:03:07.363873 22262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:03:07.364152 22262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:03:07.365288 22262 hybrid_clock.cc:648] HybridClock initialized: now 1790226187365269 us; error 37 us; skew 500 ppm
Sep 24 05:03:07 dist-test-slave-70st krb5kdc[22249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226187, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:03:07.373945 22262 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 05:03:07.374389 22262 webserver.cc:460] Webserver started at http://127.0.98.62:36403/ using document root <none> and password file <none>
I20260924 05:03:07.374617 22262 fs_manager.cc:359] Metadata directory not provided
I20260924 05:03:07.374666 22262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:03:07.374789 22262 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:03:07.375818 22262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790225929848825-392-0/minicluster-data/master-0/data/instance:
uuid: "4195964eb8da40d7b74072a0207bdf33"
format_stamp: "Formatted at 2026-09-24 05:03:07 on dist-test-slave-70st"
server_key: "05afcb35d0a8f163d7455694f917aa03"
server_key_iv: "34d5aca3996ac61e40e685ef47ee794d"
server_key_version: "encryptionkey@0"
I20260924 05:03:07.376132 22262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790225929848825-392-0/minicluster-data/master-0/wal/instance:
uuid: "4195964eb8da40d7b74072a0207bdf33"
format_stamp: "Formatted at 2026-09-24 05:03:07 on dist-test-slave-70st"
server_key: "05afcb35d0a8f163d7455694f917aa03"
server_key_iv: "34d5aca3996ac61e40e685ef47ee794d"
server_key_version: "encryptionkey@0"
I20260924 05:03:07.377332 22262 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260924 05:03:07.378125 22277 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:03:07.378284 22262 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 05:03:07.378356 22262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790225929848825-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790225929848825-392-0/minicluster-data/master-0/wal
uuid: "4195964eb8da40d7b74072a0207bdf33"
format_stamp: "Formatted at 2026-09-24 05:03:07 on dist-test-slave-70st"
server_key: "05afcb35d0a8f163d7455694f917aa03"
server_key_iv: "34d5aca3996ac61e40e685ef47ee794d"
server_key_version: "encryptionkey@0"
I20260924 05:03:07.378419 22262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790225929848825-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790225929848825-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790225929848825-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:03:07.392334 22262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:03:07.394431 22262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:03:07.394574 22262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:03:07.398413 22262 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45519
I20260924 05:03:07.398452 22329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45519 every 8 connection(s)
I20260924 05:03:07.398769 22262 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:03:07.399611 22330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:03:07.401835 22330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4195964eb8da40d7b74072a0207bdf33: Bootstrap starting.
I20260924 05:03:07.402107 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 22262
I20260924 05:03:07.402205 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790225929848825-392-0/minicluster-data/master-0/wal/instance
I20260924 05:03:07.402334 392 external_mini_cluster.cc:1428] Setting key 2f85e11ffa82db49fd6f7cbed33d8029
I20260924 05:03:07.402419 22330 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4195964eb8da40d7b74072a0207bdf33: Neither blocks nor log segments found. Creating new log.
I20260924 05:03:07.402776 22330 log.cc:824] T 00000000000000000000000000000000 P 4195964eb8da40d7b74072a0207bdf33: Log is configured to *not* fsync() on all Append() calls
I20260924 05:03:07.403398 22330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4195964eb8da40d7b74072a0207bdf33: No bootstrap required, opened a new log
I20260924 05:03:07.405153 22330 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4195964eb8da40d7b74072a0207bdf33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4195964eb8da40d7b74072a0207bdf33" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45519 } }
I20260924 05:03:07.405305 22330 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4195964eb8da40d7b74072a0207bdf33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:03:07.405339 22330 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4195964eb8da40d7b74072a0207bdf33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4195964eb8da40d7b74072a0207bdf33, State: Initialized, Role: FOLLOWER
I20260924 05:03:07.405494 22330 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4195964eb8da40d7b74072a0207bdf33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4195964eb8da40d7b74072a0207bdf33" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45519 } }
I20260924 05:03:07.405568 22330 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4195964eb8da40d7b74072a0207bdf33 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:03:07.405598 22330 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4195964eb8da40d7b74072a0207bdf33 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:03:07.405644 22330 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4195964eb8da40d7b74072a0207bdf33 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:03:07.406153 22330 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4195964eb8da40d7b74072a0207bdf33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4195964eb8da40d7b74072a0207bdf33" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45519 } }
I20260924 05:03:07.406244 22330 leader_election.cc:302] T 00000000000000000000000000000000 P 4195964eb8da40d7b74072a0207bdf33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4195964eb8da40d7b74072a0207bdf33; no voters:
I20260924 05:03:07.406391 22330 leader_election.cc:288] T 00000000000000000000000000000000 P 4195964eb8da40d7b74072a0207bdf33 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:03:07.406438 22335 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4195964eb8da40d7b74072a0207bdf33 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:03:07.406625 22330 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4195964eb8da40d7b74072a0207bdf33 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:03:07.406651 22335 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4195964eb8da40d7b74072a0207bdf33 [term 1 LEADER]: Becoming Leader. State: Replica: 4195964eb8da40d7b74072a0207bdf33, State: Running, Role: LEADER
I20260924 05:03:07.406790 22335 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4195964eb8da40d7b74072a0207bdf33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4195964eb8da40d7b74072a0207bdf33" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45519 } }
I20260924 05:03:07.407162 22337 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4195964eb8da40d7b74072a0207bdf33 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4195964eb8da40d7b74072a0207bdf33. Latest consensus state: current_term: 1 leader_uuid: "4195964eb8da40d7b74072a0207bdf33" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4195964eb8da40d7b74072a0207bdf33" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45519 } } }
I20260924 05:03:07.407184 22336 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4195964eb8da40d7b74072a0207bdf33 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4195964eb8da40d7b74072a0207bdf33" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4195964eb8da40d7b74072a0207bdf33" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45519 } } }
I20260924 05:03:07.407315 22337 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4195964eb8da40d7b74072a0207bdf33 [sys.catalog]: This master's current role is: LEADER
I20260924 05:03:07.407330 22336 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4195964eb8da40d7b74072a0207bdf33 [sys.catalog]: This master's current role is: LEADER
I20260924 05:03:07.407544 22344 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:03:07.408058 22344 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 24 05:03:07 dist-test-slave-70st krb5kdc[22249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226187, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:03:07.409781 22344 catalog_manager.cc:1347] Generated new cluster ID: 975a4f0109154f73b626c0e714eeab39
I20260924 05:03:07.409861 22344 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:03:07.413236 22333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:07.402787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56897 (local address 127.0.98.62:45519)
0924 05:03:07.402997 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:07.403007 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:07.403034 (+ 27us) server_negotiation.cc:407] Connection header received
0924 05:03:07.403438 (+ 404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:07.403444 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:07.403505 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:07.403642 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:03:07.404229 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:07.405119 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:07.405992 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:07.406495 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:07.406942 (+ 447us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:07.409546 (+ 2604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:03:07.409550 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:03:07.409550 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:03:07.409561 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:03:07.411435 (+ 1874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:07.411820 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:07.411822 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:07.411823 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:07.411914 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:07.412100 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:07.412101 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:07.412101 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:07.412399 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:03:07.412541 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0924 05:03:07.412654 (+ 113us) server_negotiation.cc:299] Negotiation successful
0924 05:03:07.412872 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":141,"thread_start_us":70,"threads_started":1}
I20260924 05:03:07.414317 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 05:03:07.419994 22344 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:03:07.420570 22344 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:03:07.422602 22333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:07.415794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52367 (local address 127.0.98.62:45519)
0924 05:03:07.415823 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:07.415825 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:07.415912 (+ 87us) server_negotiation.cc:407] Connection header received
0924 05:03:07.415983 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:07.415984 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:07.415992 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:07.416040 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:03:07.416613 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:07.417319 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:07.418331 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:07.418643 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:07.418698 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:07.420028 (+ 1330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:03:07.420030 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:03:07.420030 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:03:07.420037 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:03:07.421554 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:07.421897 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:07.421898 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:07.421898 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:07.421962 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:07.422124 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:07.422125 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:07.422125 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:07.422242 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:03:07.422333 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 05:03:07.422428 (+ 95us) server_negotiation.cc:299] Negotiation successful
0924 05:03:07.422511 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13}
I20260924 05:03:07.424019 22344 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4195964eb8da40d7b74072a0207bdf33: Generated new TSK 0
I20260924 05:03:07.424175 22344 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 05:03:07.429339 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 22262
2026-09-24T05:03:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1205 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790225929848825-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:03:07 dist-test-slave-70st krb5kdc[22361](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:03:07 dist-test-slave-70st krb5kdc[22361](info): set up 2 sockets
Sep 24 05:03:07 dist-test-slave-70st krb5kdc[22361](info): commencing operation
krb5kdc: starting...
W20260924 05:03:08.389191 392 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790225929848825-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790225929848825-392-0/krb5kdc/test-user.keytab.
Sep 24 05:03:08 dist-test-slave-70st krb5kdc[22361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226188, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:03:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:03:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 05:03:08.500527 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790225929848825-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790225929848825-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790225929848825-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42419
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42419
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:03:08.578032 22374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:03:08.578235 22374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:03:08.578266 22374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:03:08.579389 22374 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:03:08.579432 22374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:03:08.579449 22374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:03:08.579464 22374 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:03:08.579478 22374 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:03:08.580916 22374 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790225929848825-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790225929848825-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42419
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22374
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790225929848825-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:03:08.581151 22374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:03:08.581352 22374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:03:08.586033 22379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:03:08.586119 22374 server_base.cc:1034] running on GCE node
W20260924 05:03:08.586050 22380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:03:08.586038 22382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:03:08.586498 22374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:03:08.586753 22374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:03:08.587898 22374 hybrid_clock.cc:648] HybridClock initialized: now 1790226188587884 us; error 29 us; skew 500 ppm
Sep 24 05:03:08 dist-test-slave-70st krb5kdc[22361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226188, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:03:08.590267 22374 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 05:03:08.590670 22374 webserver.cc:460] Webserver started at http://127.0.98.62:35079/ using document root <none> and password file <none>
I20260924 05:03:08.590862 22374 fs_manager.cc:359] Metadata directory not provided
I20260924 05:03:08.590906 22374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:03:08.591022 22374 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:03:08.592046 22374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790225929848825-392-0/minicluster-data/master-0/data/instance:
uuid: "5ed5ad2fa8b1428aa8b64c03fa900d55"
format_stamp: "Formatted at 2026-09-24 05:03:08 on dist-test-slave-70st"
server_key: "f836133ff62dbb233825fda27f3bb8b4"
server_key_iv: "eb9e7bec44407a32a7f3d1335961163d"
server_key_version: "encryptionkey@0"
I20260924 05:03:08.592361 22374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790225929848825-392-0/minicluster-data/master-0/wal/instance:
uuid: "5ed5ad2fa8b1428aa8b64c03fa900d55"
format_stamp: "Formatted at 2026-09-24 05:03:08 on dist-test-slave-70st"
server_key: "f836133ff62dbb233825fda27f3bb8b4"
server_key_iv: "eb9e7bec44407a32a7f3d1335961163d"
server_key_version: "encryptionkey@0"
I20260924 05:03:08.593585 22374 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 05:03:08.594237 22389 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:03:08.594381 22374 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 05:03:08.594477 22374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790225929848825-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790225929848825-392-0/minicluster-data/master-0/wal
uuid: "5ed5ad2fa8b1428aa8b64c03fa900d55"
format_stamp: "Formatted at 2026-09-24 05:03:08 on dist-test-slave-70st"
server_key: "f836133ff62dbb233825fda27f3bb8b4"
server_key_iv: "eb9e7bec44407a32a7f3d1335961163d"
server_key_version: "encryptionkey@0"
I20260924 05:03:08.594540 22374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790225929848825-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790225929848825-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790225929848825-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:03:08.602886 22374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:03:08.604976 22374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:03:08.605108 22374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:03:08.608708 22374 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42419
I20260924 05:03:08.608752 22441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42419 every 8 connection(s)
I20260924 05:03:08.609063 22374 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:03:08.609892 22442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:03:08.612046 22442 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ed5ad2fa8b1428aa8b64c03fa900d55: Bootstrap starting.
I20260924 05:03:08.612546 22442 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5ed5ad2fa8b1428aa8b64c03fa900d55: Neither blocks nor log segments found. Creating new log.
I20260924 05:03:08.612768 22442 log.cc:824] T 00000000000000000000000000000000 P 5ed5ad2fa8b1428aa8b64c03fa900d55: Log is configured to *not* fsync() on all Append() calls
I20260924 05:03:08.613379 22442 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ed5ad2fa8b1428aa8b64c03fa900d55: No bootstrap required, opened a new log
I20260924 05:03:08.614547 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 22374
I20260924 05:03:08.614642 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790225929848825-392-0/minicluster-data/master-0/wal/instance
I20260924 05:03:08.614796 392 external_mini_cluster.cc:1428] Setting key d21c3915dc079109120fd7885511929e
I20260924 05:03:08.615057 22442 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ed5ad2fa8b1428aa8b64c03fa900d55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ed5ad2fa8b1428aa8b64c03fa900d55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42419 } }
I20260924 05:03:08.615238 22442 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5ed5ad2fa8b1428aa8b64c03fa900d55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:03:08.615274 22442 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ed5ad2fa8b1428aa8b64c03fa900d55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ed5ad2fa8b1428aa8b64c03fa900d55, State: Initialized, Role: FOLLOWER
I20260924 05:03:08.615394 22442 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ed5ad2fa8b1428aa8b64c03fa900d55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ed5ad2fa8b1428aa8b64c03fa900d55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42419 } }
I20260924 05:03:08.615495 22442 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5ed5ad2fa8b1428aa8b64c03fa900d55 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:03:08.615533 22442 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5ed5ad2fa8b1428aa8b64c03fa900d55 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:03:08.615583 22442 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ed5ad2fa8b1428aa8b64c03fa900d55 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:03:08.616505 22442 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5ed5ad2fa8b1428aa8b64c03fa900d55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ed5ad2fa8b1428aa8b64c03fa900d55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42419 } }
I20260924 05:03:08.616604 22442 leader_election.cc:302] T 00000000000000000000000000000000 P 5ed5ad2fa8b1428aa8b64c03fa900d55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5ed5ad2fa8b1428aa8b64c03fa900d55; no voters:
I20260924 05:03:08.616761 22442 leader_election.cc:288] T 00000000000000000000000000000000 P 5ed5ad2fa8b1428aa8b64c03fa900d55 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:03:08.616806 22447 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5ed5ad2fa8b1428aa8b64c03fa900d55 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:03:08.616940 22447 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5ed5ad2fa8b1428aa8b64c03fa900d55 [term 1 LEADER]: Becoming Leader. State: Replica: 5ed5ad2fa8b1428aa8b64c03fa900d55, State: Running, Role: LEADER
I20260924 05:03:08.617017 22442 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ed5ad2fa8b1428aa8b64c03fa900d55 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:03:08.617058 22447 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5ed5ad2fa8b1428aa8b64c03fa900d55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ed5ad2fa8b1428aa8b64c03fa900d55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42419 } }
I20260924 05:03:08.617369 22447 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ed5ad2fa8b1428aa8b64c03fa900d55 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ed5ad2fa8b1428aa8b64c03fa900d55. Latest consensus state: current_term: 1 leader_uuid: "5ed5ad2fa8b1428aa8b64c03fa900d55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ed5ad2fa8b1428aa8b64c03fa900d55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42419 } } }
I20260924 05:03:08.617398 22448 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ed5ad2fa8b1428aa8b64c03fa900d55 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5ed5ad2fa8b1428aa8b64c03fa900d55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ed5ad2fa8b1428aa8b64c03fa900d55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42419 } } }
I20260924 05:03:08.617501 22447 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ed5ad2fa8b1428aa8b64c03fa900d55 [sys.catalog]: This master's current role is: LEADER
I20260924 05:03:08.617508 22448 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ed5ad2fa8b1428aa8b64c03fa900d55 [sys.catalog]: This master's current role is: LEADER
I20260924 05:03:08.617708 22454 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:03:08.618340 22454 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:03:08.620278 22446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:08.615333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59707 (local address 127.0.98.62:42419)
0924 05:03:08.615576 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:08.615586 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:08.615617 (+ 31us) server_negotiation.cc:407] Connection header received
0924 05:03:08.615731 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:08.615737 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:08.615781 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:08.615884 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:03:08.616158 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:08.617243 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:08.618374 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:08.618874 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:08.618972 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:08.619631 (+ 659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:03:08.619635 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:03:08.619635 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:03:08.619803 (+ 168us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:03:08.619832 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:03:08.619843 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:03:08.619895 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 05:03:08.620001 (+ 106us) server_negotiation.cc:299] Negotiation successful
0924 05:03:08.620066 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":117,"threads_started":1}
I20260924 05:03:08.620396 22454 catalog_manager.cc:1347] Generated new cluster ID: 08797b72d2094407907d11f290c728a7
I20260924 05:03:08.620466 22454 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:03:08.621080 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 05:03:08.628253 22446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:08.625743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34389 (local address 127.0.98.62:42419)
0924 05:03:08.625789 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:08.625791 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:08.625920 (+ 129us) server_negotiation.cc:407] Connection header received
0924 05:03:08.625981 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:08.625982 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:08.625992 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:08.626042 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:03:08.626154 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:08.626604 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:08.627247 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:08.627424 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:08.627482 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:08.628043 (+ 561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:03:08.628044 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:03:08.628044 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:03:08.628048 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:03:08.628063 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:03:08.628067 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:03:08.628110 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 05:03:08.628157 (+ 47us) server_negotiation.cc:299] Negotiation successful
0924 05:03:08.628181 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 05:03:08.637072 22454 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:03:08.637846 22454 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:03:08.641726 22454 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5ed5ad2fa8b1428aa8b64c03fa900d55: Generated new TSK 0
I20260924 05:03:08.641907 22454 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 05:03:08.643683 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 22374
2026-09-24T05:03:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1213 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790225929848825-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:03:08 dist-test-slave-70st krb5kdc[22473](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:03:08 dist-test-slave-70st krb5kdc[22473](info): set up 2 sockets
Sep 24 05:03:08 dist-test-slave-70st krb5kdc[22473](info): commencing operation
krb5kdc: starting...
W20260924 05:03:09.589236 392 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790225929848825-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790225929848825-392-0/krb5kdc/test-user.keytab.
Sep 24 05:03:09 dist-test-slave-70st krb5kdc[22473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226189, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:03:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:03:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 05:03:09.699869 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790225929848825-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790225929848825-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790225929848825-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44657
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44657
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:03:09.777957 22486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:03:09.778188 22486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:03:09.778234 22486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:03:09.779385 22486 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:03:09.779433 22486 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:03:09.779456 22486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:03:09.779475 22486 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:03:09.779495 22486 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:03:09.780990 22486 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790225929848825-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790225929848825-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44657
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22486
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790225929848825-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:03:09.781248 22486 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:03:09.781487 22486 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 05:03:09.786219 22486 server_base.cc:1034] running on GCE node
W20260924 05:03:09.786180 22491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:03:09.786201 22492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:03:09.786180 22494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:03:09.786574 22486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:03:09.786820 22486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:03:09.787973 22486 hybrid_clock.cc:648] HybridClock initialized: now 1790226189787957 us; error 34 us; skew 500 ppm
Sep 24 05:03:09 dist-test-slave-70st krb5kdc[22473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226189, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:03:09.790232 22486 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 05:03:09.790638 22486 webserver.cc:460] Webserver started at http://127.0.98.62:36903/ using document root <none> and password file <none>
I20260924 05:03:09.790832 22486 fs_manager.cc:359] Metadata directory not provided
I20260924 05:03:09.790879 22486 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:03:09.791006 22486 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:03:09.792146 22486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790225929848825-392-0/minicluster-data/master-0/data/instance:
uuid: "75fcfc25dda34e5f9ec12d81c1fdc0ee"
format_stamp: "Formatted at 2026-09-24 05:03:09 on dist-test-slave-70st"
server_key: "9dc9058b8fee96ac7df10364c27eee6e"
server_key_iv: "24eadce7add7e1179ac75c3b4328908e"
server_key_version: "encryptionkey@0"
I20260924 05:03:09.792516 22486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790225929848825-392-0/minicluster-data/master-0/wal/instance:
uuid: "75fcfc25dda34e5f9ec12d81c1fdc0ee"
format_stamp: "Formatted at 2026-09-24 05:03:09 on dist-test-slave-70st"
server_key: "9dc9058b8fee96ac7df10364c27eee6e"
server_key_iv: "24eadce7add7e1179ac75c3b4328908e"
server_key_version: "encryptionkey@0"
I20260924 05:03:09.793751 22486 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 05:03:09.794498 22501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:03:09.794642 22486 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 05:03:09.794724 22486 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790225929848825-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790225929848825-392-0/minicluster-data/master-0/wal
uuid: "75fcfc25dda34e5f9ec12d81c1fdc0ee"
format_stamp: "Formatted at 2026-09-24 05:03:09 on dist-test-slave-70st"
server_key: "9dc9058b8fee96ac7df10364c27eee6e"
server_key_iv: "24eadce7add7e1179ac75c3b4328908e"
server_key_version: "encryptionkey@0"
I20260924 05:03:09.794782 22486 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790225929848825-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790225929848825-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790225929848825-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:03:09.797906 22486 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:03:09.799585 22486 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:03:09.799701 22486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:03:09.803453 22486 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44657
I20260924 05:03:09.803537 22553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44657 every 8 connection(s)
I20260924 05:03:09.803802 22486 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:03:09.803934 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 22486
I20260924 05:03:09.804032 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790225929848825-392-0/minicluster-data/master-0/wal/instance
I20260924 05:03:09.804199 392 external_mini_cluster.cc:1428] Setting key b7e32fa1a5c4bc8657db294ee854c444
I20260924 05:03:09.804857 22554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:03:09.806654 22557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:09.804704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44655 (local address 127.0.98.62:44657)
0924 05:03:09.804892 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:09.804901 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:09.804931 (+ 30us) server_negotiation.cc:407] Connection header received
0924 05:03:09.805510 (+ 579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:09.805518 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:09.805552 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:09.805704 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:03:09.806251 (+ 547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:03:09.806255 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:03:09.806255 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:03:09.806259 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:03:09.806304 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:03:09.806326 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:03:09.806401 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 05:03:09.806438 (+ 37us) server_negotiation.cc:299] Negotiation successful
0924 05:03:09.806476 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":84,"threads_started":1}
I20260924 05:03:09.807574 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 05:03:09.807899 22554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75fcfc25dda34e5f9ec12d81c1fdc0ee: Bootstrap starting.
I20260924 05:03:09.808558 22554 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 75fcfc25dda34e5f9ec12d81c1fdc0ee: Neither blocks nor log segments found. Creating new log.
I20260924 05:03:09.808887 22554 log.cc:824] T 00000000000000000000000000000000 P 75fcfc25dda34e5f9ec12d81c1fdc0ee: Log is configured to *not* fsync() on all Append() calls
I20260924 05:03:09.809573 22554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75fcfc25dda34e5f9ec12d81c1fdc0ee: No bootstrap required, opened a new log
I20260924 05:03:09.811173 22554 raft_consensus.cc:348] T 00000000000000000000000000000000 P 75fcfc25dda34e5f9ec12d81c1fdc0ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75fcfc25dda34e5f9ec12d81c1fdc0ee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44657 } }
I20260924 05:03:09.811303 22554 raft_consensus.cc:374] T 00000000000000000000000000000000 P 75fcfc25dda34e5f9ec12d81c1fdc0ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 05:03:09.811332 22554 raft_consensus.cc:729] T 00000000000000000000000000000000 P 75fcfc25dda34e5f9ec12d81c1fdc0ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75fcfc25dda34e5f9ec12d81c1fdc0ee, State: Initialized, Role: FOLLOWER
I20260924 05:03:09.811424 22554 consensus_queue.cc:260] T 00000000000000000000000000000000 P 75fcfc25dda34e5f9ec12d81c1fdc0ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75fcfc25dda34e5f9ec12d81c1fdc0ee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44657 } }
I20260924 05:03:09.811512 22554 raft_consensus.cc:388] T 00000000000000000000000000000000 P 75fcfc25dda34e5f9ec12d81c1fdc0ee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 05:03:09.811550 22554 raft_consensus.cc:482] T 00000000000000000000000000000000 P 75fcfc25dda34e5f9ec12d81c1fdc0ee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 05:03:09.811592 22554 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 75fcfc25dda34e5f9ec12d81c1fdc0ee [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:03:09.812219 22554 raft_consensus.cc:504] T 00000000000000000000000000000000 P 75fcfc25dda34e5f9ec12d81c1fdc0ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75fcfc25dda34e5f9ec12d81c1fdc0ee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44657 } }
I20260924 05:03:09.812337 22554 leader_election.cc:302] T 00000000000000000000000000000000 P 75fcfc25dda34e5f9ec12d81c1fdc0ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 75fcfc25dda34e5f9ec12d81c1fdc0ee; no voters:
I20260924 05:03:09.812484 22554 leader_election.cc:288] T 00000000000000000000000000000000 P 75fcfc25dda34e5f9ec12d81c1fdc0ee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 05:03:09.812528 22564 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 75fcfc25dda34e5f9ec12d81c1fdc0ee [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:03:09.812686 22554 sys_catalog.cc:564] T 00000000000000000000000000000000 P 75fcfc25dda34e5f9ec12d81c1fdc0ee [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:03:09.812700 22564 raft_consensus.cc:686] T 00000000000000000000000000000000 P 75fcfc25dda34e5f9ec12d81c1fdc0ee [term 1 LEADER]: Becoming Leader. State: Replica: 75fcfc25dda34e5f9ec12d81c1fdc0ee, State: Running, Role: LEADER
I20260924 05:03:09.812778 22564 consensus_queue.cc:237] T 00000000000000000000000000000000 P 75fcfc25dda34e5f9ec12d81c1fdc0ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75fcfc25dda34e5f9ec12d81c1fdc0ee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44657 } }
I20260924 05:03:09.813149 22566 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75fcfc25dda34e5f9ec12d81c1fdc0ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75fcfc25dda34e5f9ec12d81c1fdc0ee. Latest consensus state: current_term: 1 leader_uuid: "75fcfc25dda34e5f9ec12d81c1fdc0ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75fcfc25dda34e5f9ec12d81c1fdc0ee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44657 } } }
I20260924 05:03:09.813248 22566 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75fcfc25dda34e5f9ec12d81c1fdc0ee [sys.catalog]: This master's current role is: LEADER
I20260924 05:03:09.813138 22565 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75fcfc25dda34e5f9ec12d81c1fdc0ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "75fcfc25dda34e5f9ec12d81c1fdc0ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75fcfc25dda34e5f9ec12d81c1fdc0ee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44657 } } }
I20260924 05:03:09.813511 22572 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:03:09.813517 22565 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75fcfc25dda34e5f9ec12d81c1fdc0ee [sys.catalog]: This master's current role is: LEADER
I20260924 05:03:09.814033 22572 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:03:09.815405 22572 catalog_manager.cc:1347] Generated new cluster ID: d19e8a1891e74bc59f25b85781af1b09
I20260924 05:03:09.815455 22572 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:03:09.818995 22557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:09.809028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47963 (local address 127.0.98.62:44657)
0924 05:03:09.809059 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:09.809061 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:09.809221 (+ 160us) server_negotiation.cc:407] Connection header received
0924 05:03:09.811471 (+ 2250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:09.811472 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:09.811477 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:09.811538 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:03:09.818787 (+ 7249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:03:09.818789 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:03:09.818789 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 05:03:09.818794 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:03:09.818808 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 05:03:09.818813 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:03:09.818861 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 05:03:09.818909 (+ 48us) server_negotiation.cc:299] Negotiation successful
0924 05:03:09.818923 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 05:03:09.823102 22572 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:03:09.823246 22572 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:03:09.827678 22572 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 75fcfc25dda34e5f9ec12d81c1fdc0ee: Generated new TSK 0
I20260924 05:03:09.827860 22572 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 05:03:09.836534 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 22486
2026-09-24T05:03:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1193 ms)
[----------] 18 tests from AuthTokenIssuingTest (21780 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225929848825-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:03:09 dist-test-slave-70st krb5kdc[22585](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:03:09 dist-test-slave-70st krb5kdc[22585](info): set up 2 sockets
Sep 24 05:03:09 dist-test-slave-70st krb5kdc[22585](info): commencing operation
krb5kdc: starting...
W20260924 05:03:10.789167 392 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225929848825-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225929848825-392-0/krb5kdc/test-user.keytab.
Sep 24 05:03:10 dist-test-slave-70st krb5kdc[22585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226190, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:03:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:03:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 05:03:10.896713 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225929848825-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225929848825-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225929848825-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44577
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44577,127.0.98.61:40007,127.0.98.60:40015
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:03:10.973762 22598 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 05:03:10.973970 22598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:03:10.973985 22598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:03:10.974007 22598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:03:10.975172 22598 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:03:10.975214 22598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:03:10.975230 22598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:03:10.975250 22598 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:03:10.975274 22598 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:03:10.976758 22598 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225929848825-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225929848825-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44577,127.0.98.61:40007,127.0.98.60:40015
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44577
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22598
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225929848825-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:03:10.976989 22598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:03:10.977187 22598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 05:03:10.981909 22598 server_base.cc:1034] running on GCE node
W20260924 05:03:10.981871 22603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:03:10.981915 22604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:03:10.981878 22606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:03:10.982434 22598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:03:10.982692 22598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:03:10.983836 22598 hybrid_clock.cc:648] HybridClock initialized: now 1790226190983823 us; error 29 us; skew 500 ppm
Sep 24 05:03:10 dist-test-slave-70st krb5kdc[22585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226190, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:03:10.986631 22598 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 05:03:10.987035 22598 webserver.cc:460] Webserver started at http://127.0.98.62:45095/ using document root <none> and password file <none>
I20260924 05:03:10.987217 22598 fs_manager.cc:359] Metadata directory not provided
I20260924 05:03:10.987262 22598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:03:10.987380 22598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:03:10.988391 22598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225929848825-392-0/minicluster-data/master-0/data/instance:
uuid: "44b689b40d724231be89dd39bbf456cf"
format_stamp: "Formatted at 2026-09-24 05:03:10 on dist-test-slave-70st"
server_key: "cb04286e4467c94f0d4c5f6ea606c04a"
server_key_iv: "94709e4abab0c1887d6d31908d1333ca"
server_key_version: "encryptionkey@0"
I20260924 05:03:10.988703 22598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225929848825-392-0/minicluster-data/master-0/wal/instance:
uuid: "44b689b40d724231be89dd39bbf456cf"
format_stamp: "Formatted at 2026-09-24 05:03:10 on dist-test-slave-70st"
server_key: "cb04286e4467c94f0d4c5f6ea606c04a"
server_key_iv: "94709e4abab0c1887d6d31908d1333ca"
server_key_version: "encryptionkey@0"
I20260924 05:03:10.989914 22598 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 05:03:10.990535 22613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:03:10.990672 22598 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:03:10.990749 22598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225929848825-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225929848825-392-0/minicluster-data/master-0/wal
uuid: "44b689b40d724231be89dd39bbf456cf"
format_stamp: "Formatted at 2026-09-24 05:03:10 on dist-test-slave-70st"
server_key: "cb04286e4467c94f0d4c5f6ea606c04a"
server_key_iv: "94709e4abab0c1887d6d31908d1333ca"
server_key_version: "encryptionkey@0"
I20260924 05:03:10.990819 22598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225929848825-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225929848825-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225929848825-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:03:11.002256 22598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:03:11.006022 22598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:03:11.006156 22598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:03:11.009899 22598 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44577
I20260924 05:03:11.009929 22665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44577 every 8 connection(s)
I20260924 05:03:11.010274 22598 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:03:11.010629 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 22598
I20260924 05:03:11.010739 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225929848825-392-0/minicluster-data/master-0/wal/instance
I20260924 05:03:11.010928 392 external_mini_cluster.cc:1428] Setting key e12e02446e4de365276675448c2cea60
I20260924 05:03:11.011400 22666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:03:11.013144 22666 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44577 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 24 05:03:11 dist-test-slave-70st krb5kdc[22585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226190, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:03:11.026208 22670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:11.014048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37350 (local address 127.0.98.62:44577)
0924 05:03:11.014211 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:11.014215 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:11.014248 (+ 33us) server_negotiation.cc:407] Connection header received
0924 05:03:11.014674 (+ 426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:11.014676 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:11.014698 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:11.014771 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:03:11.015298 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:11.016081 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:11.016905 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:11.017193 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:11.017284 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:11.020965 (+ 3681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:03:11.020973 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:03:11.020974 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:03:11.020982 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:03:11.024506 (+ 3524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:11.024978 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:11.024981 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:11.024982 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:11.025089 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:11.025340 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:11.025341 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:11.025341 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:11.025605 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:03:11.025667 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 05:03:11.025886 (+ 219us) server_negotiation.cc:299] Negotiation successful
0924 05:03:11.026017 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":17,"server-negotiator.queue_time_us":136,"thread_start_us":63,"threads_started":1}
I20260924 05:03:11.026212 22669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:11.013858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44577 (local address 127.0.0.1:37350)
0924 05:03:11.014016 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 05:03:11.014049 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 05:03:11.014534 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:03:11.014784 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:03:11.014787 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:03:11.015063 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:03:11.015207 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:03:11.015211 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:03:11.016186 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:03:11.016186 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:03:11.016833 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:03:11.016835 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:03:11.017276 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:03:11.017277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:03:11.017335 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:11.018406 (+ 1071us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:03:11.018416 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:03:11.020853 (+ 2437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:03:11.024588 (+ 3735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:03:11.024596 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:03:11.024600 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:03:11.024877 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:03:11.025162 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:03:11.025163 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:03:11.025164 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:03:11.025276 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:03:11.025665 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:03:11.025672 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:03:11.025749 (+ 77us) client_negotiation.cc:769] Sending connection context
0924 05:03:11.025869 (+ 120us) client_negotiation.cc:240] Negotiation successful
0924 05:03:11.026079 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"mutex_wait_us":53,"thread_start_us":79,"threads_started":1}
I20260924 05:03:11.026966 22666 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40007 } has no permanent_uuid. Determining permanent_uuid...
I20260924 05:03:11.027318 22669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:11.027201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40007 (local address 127.0.0.1:45864)
0924 05:03:11.027249 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 05:03:11.027296 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260924 05:03:11.027426 22617 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40007: connect: Connection refused (error 111)
I20260924 05:03:11.027791 22669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:11.027710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40007 (local address 127.0.0.1:45870)
0924 05:03:11.027750 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0924 05:03:11.027774 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260924 05:03:11.027889 22666 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:40007: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40007: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225929848825-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225929848825-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260924 05:03:11.035516 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225929848825-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225929848825-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225929848825-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225929848825-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:40007
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225929848825-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44577,127.0.98.61:40007,127.0.98.60:40015
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 05:03:11.077103 22666 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40007 } attempt: 1
I20260924 05:03:11.077592 22669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:11.077479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40007 (local address 127.0.0.1:45884)
0924 05:03:11.077530 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0924 05:03:11.077562 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260924 05:03:11.077880 22669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:11.077807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40007 (local address 127.0.0.1:45894)
0924 05:03:11.077839 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0924 05:03:11.077863 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260924 05:03:11.077963 22666 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:40007: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40007: connect: Connection refused (error 111)
W20260924 05:03:11.114269 22673 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 05:03:11.114496 22673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:03:11.114521 22673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:03:11.114544 22673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:03:11.115669 22673 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:03:11.115713 22673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:03:11.115731 22673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:03:11.115747 22673 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:03:11.115760 22673 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:03:11.117235 22673 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225929848825-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225929848825-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44577,127.0.98.61:40007,127.0.98.60:40015
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225929848825-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:40007
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225929848825-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22673
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225929848825-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:03:11.117502 22673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:03:11.117702 22673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:03:11.122320 22681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:03:11.122316 22678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:03:11.122390 22679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:03:11.122529 22673 server_base.cc:1034] running on GCE node
I20260924 05:03:11.122766 22673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:03:11.123066 22673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:03:11.124202 22673 hybrid_clock.cc:648] HybridClock initialized: now 1790226191124192 us; error 27 us; skew 500 ppm
Sep 24 05:03:11 dist-test-slave-70st krb5kdc[22585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226191, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:03:11.126986 22673 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260924 05:03:11.127379 22673 webserver.cc:460] Webserver started at http://127.0.98.61:37331/ using document root <none> and password file <none>
I20260924 05:03:11.127583 22673 fs_manager.cc:359] Metadata directory not provided
I20260924 05:03:11.127629 22673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:03:11.127749 22673 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:03:11.128757 22673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225929848825-392-0/minicluster-data/master-1/data/instance:
uuid: "b7fb492c0e5f49218d8aa48b49192e48"
format_stamp: "Formatted at 2026-09-24 05:03:11 on dist-test-slave-70st"
server_key: "bb7650af36ad3925bc953c34b8a6dafb"
server_key_iv: "d95f6b053a7017a04edca125e692e3cf"
server_key_version: "encryptionkey@0"
I20260924 05:03:11.129078 22673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225929848825-392-0/minicluster-data/master-1/wal/instance:
uuid: "b7fb492c0e5f49218d8aa48b49192e48"
format_stamp: "Formatted at 2026-09-24 05:03:11 on dist-test-slave-70st"
server_key: "bb7650af36ad3925bc953c34b8a6dafb"
server_key_iv: "d95f6b053a7017a04edca125e692e3cf"
server_key_version: "encryptionkey@0"
I20260924 05:03:11.130293 22673 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 05:03:11.130906 22688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:03:11.131054 22673 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 05:03:11.131134 22673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225929848825-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225929848825-392-0/minicluster-data/master-1/wal
uuid: "b7fb492c0e5f49218d8aa48b49192e48"
format_stamp: "Formatted at 2026-09-24 05:03:11 on dist-test-slave-70st"
server_key: "bb7650af36ad3925bc953c34b8a6dafb"
server_key_iv: "d95f6b053a7017a04edca125e692e3cf"
server_key_version: "encryptionkey@0"
I20260924 05:03:11.131189 22673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225929848825-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225929848825-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225929848825-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:03:11.141862 22673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:03:11.144536 22673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:03:11.144660 22673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:03:11.146106 22666 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40007 } attempt: 2
I20260924 05:03:11.146524 22669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:11.146382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40007 (local address 127.0.0.1:45898)
0924 05:03:11.146444 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0924 05:03:11.146486 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260924 05:03:11.146879 22669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:11.146783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40007 (local address 127.0.0.1:45900)
0924 05:03:11.146827 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0924 05:03:11.146857 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260924 05:03:11.146978 22666 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:40007: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40007: connect: Connection refused (error 111)
I20260924 05:03:11.148576 22673 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:40007
I20260924 05:03:11.148607 22740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:40007 every 8 connection(s)
I20260924 05:03:11.148905 22673 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225929848825-392-0/minicluster-data/master-1/data/info.pb
I20260924 05:03:11.149821 22741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:03:11.149953 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 22673
I20260924 05:03:11.150053 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225929848825-392-0/minicluster-data/master-1/wal/instance
I20260924 05:03:11.150216 392 external_mini_cluster.cc:1428] Setting key 915c7a851c87130f96bf161e928cf0d1
I20260924 05:03:11.151813 22741 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44577 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 24 05:03:11 dist-test-slave-70st krb5kdc[22585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226191, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:03:11.164618 22670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:11.152467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37356 (local address 127.0.98.62:44577)
0924 05:03:11.152525 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:11.152528 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:11.152760 (+ 232us) server_negotiation.cc:407] Connection header received
0924 05:03:11.153328 (+ 568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:11.153329 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:11.153341 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:11.153377 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:03:11.154200 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:11.154666 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:11.156076 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:11.156253 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:11.156298 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:11.161065 (+ 4767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:03:11.161067 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:03:11.161067 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:03:11.161074 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:03:11.163397 (+ 2323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:11.163758 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:11.163759 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:11.163760 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:11.163811 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:11.164025 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:11.164026 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:11.164027 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:11.164135 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:03:11.164188 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 05:03:11.164441 (+ 253us) server_negotiation.cc:299] Negotiation successful
0924 05:03:11.164526 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 05:03:11.164796 22744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:11.152458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44577 (local address 127.0.0.1:37356)
0924 05:03:11.152634 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0924 05:03:11.152678 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 05:03:11.153239 (+ 561us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:03:11.153495 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:03:11.153499 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:03:11.153832 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:03:11.154115 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:03:11.154121 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:03:11.154757 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:03:11.154758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:03:11.156003 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:03:11.156006 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:03:11.156345 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:03:11.156346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:03:11.156497 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:11.157504 (+ 1007us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:03:11.157514 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:03:11.160973 (+ 3459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:03:11.163477 (+ 2504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:03:11.163484 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:03:11.163486 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:03:11.163694 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:03:11.163863 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:03:11.163863 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:03:11.163863 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:03:11.163973 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:03:11.164187 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:03:11.164193 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:03:11.164328 (+ 135us) client_negotiation.cc:769] Sending connection context
0924 05:03:11.164415 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 05:03:11.164576 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":76,"threads_started":1}
I20260924 05:03:11.165364 22741 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40007 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 05:03:11 dist-test-slave-70st krb5kdc[22585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226191, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 05:03:11.174216 22744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:11.165645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40007 (local address 127.0.0.1:45906)
0924 05:03:11.165684 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0924 05:03:11.165694 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:03:11.165730 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:03:11.166006 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:03:11.166007 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:03:11.166109 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:03:11.166158 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:03:11.166159 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:03:11.167347 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:03:11.167347 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:03:11.167838 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:03:11.167839 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:03:11.168283 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:03:11.168283 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:03:11.168335 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:11.169184 (+ 849us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:03:11.169188 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:03:11.170700 (+ 1512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:03:11.173376 (+ 2676us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:03:11.173377 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:03:11.173377 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:03:11.173549 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:03:11.173702 (+ 153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:03:11.173702 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:03:11.173702 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:03:11.173759 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:03:11.173982 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:03:11.173983 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:03:11.174017 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 05:03:11.174062 (+ 45us) client_negotiation.cc:240] Negotiation successful
0924 05:03:11.174137 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260924 05:03:11.174345 22746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:11.165703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45906 (local address 127.0.98.61:40007)
0924 05:03:11.165856 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:11.165861 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:11.165883 (+ 22us) server_negotiation.cc:407] Connection header received
0924 05:03:11.165905 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:11.165908 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:11.165921 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:11.166005 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:03:11.166224 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:11.167287 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:11.167909 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:11.168226 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:11.168281 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:11.170778 (+ 2497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:03:11.170784 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:03:11.170785 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:03:11.170791 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:03:11.173306 (+ 2515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:11.173602 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:11.173605 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:11.173605 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:11.173654 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:11.173806 (+ 152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:11.173807 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:11.173807 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:11.173937 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:03:11.173995 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 05:03:11.174085 (+ 90us) server_negotiation.cc:299] Negotiation successful
0924 05:03:11.174215 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":39,"server-negotiator.queue_time_us":124,"thread_start_us":85,"threads_started":1}
I20260924 05:03:11.174862 22741 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40015 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225929848825-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225929848825-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260924 05:03:11.175169 22744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:11.175068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40015 (local address 127.0.0.1:34332)
0924 05:03:11.175107 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0924 05:03:11.175149 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260924 05:03:11.175309 22689 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40015: connect: Connection refused (error 111)
I20260924 05:03:11.175675 22744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:11.175593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40015 (local address 127.0.0.1:34344)
0924 05:03:11.175631 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0924 05:03:11.175654 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260924 05:03:11.175750 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225929848825-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225929848825-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225929848825-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225929848825-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:40015
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225929848825-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44577,127.0.98.61:40007,127.0.98.60:40015
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:03:11.175783 22741 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40015: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40015: connect: Connection refused (error 111)
I20260924 05:03:11.225019 22741 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40015 } attempt: 1
I20260924 05:03:11.225565 22744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:11.225379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40015 (local address 127.0.0.1:34352)
0924 05:03:11.225471 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0924 05:03:11.225524 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260924 05:03:11.225899 22744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:11.225823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40015 (local address 127.0.0.1:34356)
0924 05:03:11.225856 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0924 05:03:11.225881 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260924 05:03:11.225997 22741 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40015: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40015: connect: Connection refused (error 111)
I20260924 05:03:11.238140 22666 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40007 } attempt: 3
Sep 24 05:03:11 dist-test-slave-70st krb5kdc[22585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226190, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 05:03:11.246598 22746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:11.238576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45920 (local address 127.0.98.61:40007)
0924 05:03:11.238643 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:11.238646 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:11.238678 (+ 32us) server_negotiation.cc:407] Connection header received
0924 05:03:11.238709 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:11.238710 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:11.238723 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:11.238777 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:03:11.239083 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:11.239820 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:11.240539 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:11.240849 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:11.240916 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:11.243246 (+ 2330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:03:11.243247 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:03:11.243248 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:03:11.243254 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:03:11.245607 (+ 2353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:11.245971 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:11.245971 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:11.245972 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:11.246018 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:11.246185 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:11.246185 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:11.246186 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:11.246279 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:03:11.246322 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 05:03:11.246471 (+ 149us) server_negotiation.cc:299] Negotiation successful
0924 05:03:11.246531 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 05:03:11.246647 22669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:11.238508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40007 (local address 127.0.0.1:45920)
0924 05:03:11.238559 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0924 05:03:11.238572 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 05:03:11.238634 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:03:11.238783 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:03:11.238784 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:03:11.238925 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:03:11.238997 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:03:11.238998 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:03:11.239904 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:03:11.239904 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:03:11.240459 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:03:11.240461 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:03:11.240917 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:03:11.240918 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:03:11.240980 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:11.241975 (+ 995us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:03:11.241981 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:03:11.243165 (+ 1184us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:03:11.245680 (+ 2515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:03:11.245680 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:03:11.245681 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:03:11.245891 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:03:11.246067 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:03:11.246068 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:03:11.246069 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:03:11.246135 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:03:11.246329 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:03:11.246329 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:03:11.246379 (+ 50us) client_negotiation.cc:769] Sending connection context
0924 05:03:11.246453 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 05:03:11.246552 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260924 05:03:11.247025 22666 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40015 } has no permanent_uuid. Determining permanent_uuid...
I20260924 05:03:11.247347 22669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:11.247236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40015 (local address 127.0.0.1:34364)
0924 05:03:11.247283 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 05:03:11.247320 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260924 05:03:11.247613 22669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:11.247546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40015 (local address 127.0.0.1:34374)
0924 05:03:11.247569 (+ 23us) negotiation.cc:107] Waiting for socket to connect
0924 05:03:11.247594 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12}
W20260924 05:03:11.247697 22666 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40015: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40015: connect: Connection refused (error 111)
W20260924 05:03:11.255131 22748 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 05:03:11.255317 22748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:03:11.255332 22748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:03:11.255350 22748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:03:11.256413 22748 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:03:11.256451 22748 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:03:11.256464 22748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:03:11.256481 22748 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:03:11.256491 22748 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:03:11.257925 22748 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225929848825-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225929848825-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44577,127.0.98.61:40007,127.0.98.60:40015
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225929848825-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:40015
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225929848825-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22748
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225929848825-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:03:11.258105 22748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:03:11.258266 22748 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:03:11.262537 22754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:03:11.262560 22753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:03:11.262563 22756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:03:11.262763 22748 server_base.cc:1034] running on GCE node
I20260924 05:03:11.263052 22748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:03:11.263330 22748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:03:11.264488 22748 hybrid_clock.cc:648] HybridClock initialized: now 1790226191264472 us; error 32 us; skew 500 ppm
Sep 24 05:03:11 dist-test-slave-70st krb5kdc[22585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226191, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:03:11.267532 22748 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260924 05:03:11.267946 22748 webserver.cc:460] Webserver started at http://127.0.98.60:36899/ using document root <none> and password file <none>
I20260924 05:03:11.268146 22748 fs_manager.cc:359] Metadata directory not provided
I20260924 05:03:11.268195 22748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:03:11.268335 22748 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:03:11.269407 22748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225929848825-392-0/minicluster-data/master-2/data/instance:
uuid: "80ca68509f87499d87566546a3372683"
format_stamp: "Formatted at 2026-09-24 05:03:11 on dist-test-slave-70st"
server_key: "f1eb66d52df0722090cfd95e10876fe9"
server_key_iv: "2e4af1ff680838bf8d9a7abc98b4e18b"
server_key_version: "encryptionkey@0"
I20260924 05:03:11.269809 22748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225929848825-392-0/minicluster-data/master-2/wal/instance:
uuid: "80ca68509f87499d87566546a3372683"
format_stamp: "Formatted at 2026-09-24 05:03:11 on dist-test-slave-70st"
server_key: "f1eb66d52df0722090cfd95e10876fe9"
server_key_iv: "2e4af1ff680838bf8d9a7abc98b4e18b"
server_key_version: "encryptionkey@0"
I20260924 05:03:11.271060 22748 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 05:03:11.271775 22763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:03:11.271919 22748 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:03:11.272002 22748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225929848825-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225929848825-392-0/minicluster-data/master-2/wal
uuid: "80ca68509f87499d87566546a3372683"
format_stamp: "Formatted at 2026-09-24 05:03:11 on dist-test-slave-70st"
server_key: "f1eb66d52df0722090cfd95e10876fe9"
server_key_iv: "2e4af1ff680838bf8d9a7abc98b4e18b"
server_key_version: "encryptionkey@0"
I20260924 05:03:11.272073 22748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225929848825-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225929848825-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225929848825-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:03:11.278817 22666 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40015 } attempt: 1
I20260924 05:03:11.279155 22669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:11.279047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40015 (local address 127.0.0.1:34384)
0924 05:03:11.279099 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0924 05:03:11.279128 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260924 05:03:11.279433 22669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:11.279356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40015 (local address 127.0.0.1:34390)
0924 05:03:11.279388 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0924 05:03:11.279410 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260924 05:03:11.279505 22666 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40015: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40015: connect: Connection refused (error 111)
I20260924 05:03:11.287019 22748 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:03:11.290627 22748 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:03:11.290761 22748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:03:11.294133 22741 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40015 } attempt: 2
I20260924 05:03:11.294468 22744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:11.294368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40015 (local address 127.0.0.1:34396)
0924 05:03:11.294406 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0924 05:03:11.294436 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260924 05:03:11.294585 22748 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:40015
I20260924 05:03:11.294667 22815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:40015 every 8 connection(s)
I20260924 05:03:11.295034 22748 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225929848825-392-0/minicluster-data/master-2/data/info.pb
I20260924 05:03:11.295842 22816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:03:11.297916 22816 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44577 } has no permanent_uuid. Determining permanent_uuid...
I20260924 05:03:11.299891 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 22748
I20260924 05:03:11.299979 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790225929848825-392-0/minicluster-data/master-2/wal/instance
I20260924 05:03:11.300148 392 external_mini_cluster.cc:1428] Setting key dbc14cff07da580abae5f3743aad45c3
Sep 24 05:03:11 dist-test-slave-70st krb5kdc[22585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226191, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 24 05:03:11 dist-test-slave-70st krb5kdc[22585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226191, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:03:11.306704 22744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:11.294762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40015 (local address 127.0.0.1:34402)
0924 05:03:11.294796 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0924 05:03:11.294807 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 05:03:11.294859 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:03:11.296318 (+ 1459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:03:11.296320 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:03:11.296453 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:03:11.296497 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:03:11.296499 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:03:11.298049 (+ 1550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:03:11.298050 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:03:11.298704 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:03:11.298708 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:03:11.299532 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:03:11.299533 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:03:11.299587 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:11.300715 (+ 1128us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:03:11.300721 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:03:11.302242 (+ 1521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:03:11.305459 (+ 3217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:03:11.305460 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:03:11.305461 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:03:11.305661 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:03:11.305943 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:03:11.305944 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:03:11.305945 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:03:11.306014 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:03:11.306370 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:03:11.306371 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:03:11.306433 (+ 62us) client_negotiation.cc:769] Sending connection context
0924 05:03:11.306505 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 05:03:11.306609 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
Sep 24 05:03:11 dist-test-slave-70st krb5kdc[22585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226190, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 05:03:11.307240 22818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:11.294909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34402 (local address 127.0.98.60:40015)
0924 05:03:11.295489 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:11.295500 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:11.295534 (+ 34us) server_negotiation.cc:407] Connection header received
0924 05:03:11.296111 (+ 577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:11.296120 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:11.296180 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:11.296341 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:03:11.296588 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:11.297960 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:11.298801 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:11.299463 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:11.299566 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:11.302327 (+ 2761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:03:11.302334 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:03:11.302335 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:03:11.302347 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:03:11.305343 (+ 2996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:11.305757 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:11.305761 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:11.305761 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:11.305866 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:11.306071 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:11.306071 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:11.306071 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:11.306303 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:03:11.306381 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 05:03:11.306948 (+ 567us) server_negotiation.cc:299] Negotiation successful
0924 05:03:11.307077 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":507,"threads_started":1}
I20260924 05:03:11.308631 22819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:11.298582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44577 (local address 127.0.0.1:37370)
0924 05:03:11.298793 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0924 05:03:11.298836 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 05:03:11.298906 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:03:11.299031 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:03:11.299033 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:03:11.299356 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:03:11.299438 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:03:11.299440 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:03:11.300334 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:03:11.300335 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:03:11.301049 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:03:11.301052 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:03:11.301531 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:03:11.301533 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:03:11.301606 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:11.302703 (+ 1097us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:03:11.302708 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:03:11.304255 (+ 1547us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:03:11.307448 (+ 3193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:03:11.307453 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:03:11.307455 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:03:11.307655 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:03:11.307875 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:03:11.307876 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:03:11.307876 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:03:11.307938 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:03:11.308226 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:03:11.308232 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:03:11.308300 (+ 68us) client_negotiation.cc:769] Sending connection context
0924 05:03:11.308406 (+ 106us) client_negotiation.cc:240] Negotiation successful
0924 05:03:11.308517 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":77,"threads_started":1}
I20260924 05:03:11.308640 22670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:11.298645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37370 (local address 127.0.98.62:44577)
0924 05:03:11.298693 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:11.298694 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:11.298901 (+ 207us) server_negotiation.cc:407] Connection header received
0924 05:03:11.298968 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:11.298969 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:11.298979 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:11.299013 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:03:11.299513 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:11.300173 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:11.301135 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:11.301410 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:11.301504 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:11.304358 (+ 2854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:03:11.304360 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:03:11.304360 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:03:11.304368 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:03:11.307365 (+ 2997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:11.307744 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:11.307745 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:11.307745 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:11.307813 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:11.308019 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:11.308020 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:11.308021 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:11.308150 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:03:11.308241 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 05:03:11.308474 (+ 233us) server_negotiation.cc:299] Negotiation successful
0924 05:03:11.308552 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 05:03:11.309059 22816 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40007 } has no permanent_uuid. Determining permanent_uuid...
I20260924 05:03:11.309185 22741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7fb492c0e5f49218d8aa48b49192e48: Bootstrap starting.
I20260924 05:03:11.310105 22741 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b7fb492c0e5f49218d8aa48b49192e48: Neither blocks nor log segments found. Creating new log.
I20260924 05:03:11.310423 22741 log.cc:824] T 00000000000000000000000000000000 P b7fb492c0e5f49218d8aa48b49192e48: Log is configured to *not* fsync() on all Append() calls
I20260924 05:03:11.311197 22741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7fb492c0e5f49218d8aa48b49192e48: No bootstrap required, opened a new log
I20260924 05:03:11.312151 22821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:11.300607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34408 (local address 127.0.98.60:40015)
0924 05:03:11.301171 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:11.301172 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:11.301178 (+ 6us) server_negotiation.cc:407] Connection header received
0924 05:03:11.301195 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:11.301195 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:11.301204 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:11.301271 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:03:11.302009 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:11.302670 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:11.303913 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:11.304191 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:11.304247 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:11.307584 (+ 3337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:03:11.307585 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:03:11.307585 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:03:11.307591 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:03:11.311083 (+ 3492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:11.311399 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:11.311400 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:11.311400 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:11.311461 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:11.311645 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:11.311646 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:11.311647 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:11.311752 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:03:11.311879 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0924 05:03:11.311961 (+ 82us) server_negotiation.cc:299] Negotiation successful
0924 05:03:11.312028 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":82,"threads_started":1}
I20260924 05:03:11.313215 22741 raft_consensus.cc:348] T 00000000000000000000000000000000 P b7fb492c0e5f49218d8aa48b49192e48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44b689b40d724231be89dd39bbf456cf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44577 } } peers { permanent_uuid: "b7fb492c0e5f49218d8aa48b49192e48" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40007 } } peers { permanent_uuid: "80ca68509f87499d87566546a3372683" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40015 } }
I20260924 05:03:11.313310 22741 raft_consensus.cc:729] T 00000000000000000000000000000000 P b7fb492c0e5f49218d8aa48b49192e48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7fb492c0e5f49218d8aa48b49192e48, State: Initialized, Role: FOLLOWER
I20260924 05:03:11.313362 22741 consensus_queue.cc:260] T 00000000000000000000000000000000 P b7fb492c0e5f49218d8aa48b49192e48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44b689b40d724231be89dd39bbf456cf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44577 } } peers { permanent_uuid: "b7fb492c0e5f49218d8aa48b49192e48" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40007 } } peers { permanent_uuid: "80ca68509f87499d87566546a3372683" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40015 } }
I20260924 05:03:11.313669 22741 sys_catalog.cc:564] T 00000000000000000000000000000000 P b7fb492c0e5f49218d8aa48b49192e48 [sys.catalog]: configured and running, proceeding with master startup.
Sep 24 05:03:11 dist-test-slave-70st krb5kdc[22585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226191, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
W20260924 05:03:11.315411 22834 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b7fb492c0e5f49218d8aa48b49192e48: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 05:03:11.315526 22834 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 05:03:11.318040 22819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:11.309285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40007 (local address 127.0.0.1:45922)
0924 05:03:11.309321 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0924 05:03:11.309331 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:03:11.309377 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:03:11.309512 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:03:11.309513 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:03:11.309654 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:03:11.309709 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:03:11.309710 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:03:11.310494 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:03:11.310495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:03:11.311205 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:03:11.311208 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:03:11.311614 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:03:11.311615 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:03:11.311683 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:11.312664 (+ 981us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:03:11.312669 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:03:11.313940 (+ 1271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:03:11.317000 (+ 3060us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:03:11.317001 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:03:11.317001 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:03:11.317211 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:03:11.317403 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:03:11.317403 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:03:11.317404 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:03:11.317480 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:03:11.317763 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:03:11.317764 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:03:11.317804 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 05:03:11.317874 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 05:03:11.317957 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
Sep 24 05:03:11 dist-test-slave-70st krb5kdc[22585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226190, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:03:11.316182 22673 master_runner.cc:186] Error getting master registration for 127.0.98.62:44577: OK, instance_id { permanent_uuid: "44b689b40d724231be89dd39bbf456cf" instance_seqno: 1790226191008795 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 05:03:11.318444 22673 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 05:03:11.316140 22823 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7fb492c0e5f49218d8aa48b49192e48 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44b689b40d724231be89dd39bbf456cf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44577 } } peers { permanent_uuid: "b7fb492c0e5f49218d8aa48b49192e48" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40007 } } peers { permanent_uuid: "80ca68509f87499d87566546a3372683" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40015 } } }
I20260924 05:03:11.318665 22746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:11.309367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45922 (local address 127.0.98.61:40007)
0924 05:03:11.309410 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:11.309412 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:11.309440 (+ 28us) server_negotiation.cc:407] Connection header received
0924 05:03:11.309457 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:11.309458 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:11.309466 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:11.309515 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:03:11.309801 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:11.310427 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:11.311285 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:11.311552 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:11.311614 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:11.314010 (+ 2396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:03:11.314011 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:03:11.314012 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:03:11.314018 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:03:11.316911 (+ 2893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:11.317285 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:11.317285 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:11.317286 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:11.317340 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:11.317541 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:11.317542 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:11.317542 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:11.317683 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:03:11.317738 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 05:03:11.318503 (+ 765us) server_negotiation.cc:299] Negotiation successful
0924 05:03:11.318582 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 05:03:11.318717 22823 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7fb492c0e5f49218d8aa48b49192e48 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 05:03:11.319058 22816 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40015 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 05:03:11 dist-test-slave-70st krb5kdc[22585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226191, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 05:03:11.323348 22670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:11.312821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37386 (local address 127.0.98.62:44577)
0924 05:03:11.312857 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:11.312858 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:11.312900 (+ 42us) server_negotiation.cc:407] Connection header received
0924 05:03:11.312953 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:11.312954 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:11.312962 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:11.313012 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:03:11.313530 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:11.314184 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:11.315122 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:11.315391 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:11.315444 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:11.318610 (+ 3166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:03:11.318612 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:03:11.318612 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:03:11.318619 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:03:11.321791 (+ 3172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:11.322592 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:11.322594 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:11.322594 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:11.322649 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:11.322872 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:11.322873 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:11.322873 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:11.322998 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:03:11.323147 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 05:03:11.323190 (+ 43us) server_negotiation.cc:299] Negotiation successful
0924 05:03:11.323261 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 05:03:11.326951 22819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:11.319276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40015 (local address 127.0.0.1:34420)
0924 05:03:11.319312 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0924 05:03:11.319321 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:03:11.319360 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:03:11.319451 (+ 91us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:03:11.319452 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:03:11.319574 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:03:11.319603 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:03:11.319604 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:03:11.320227 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:03:11.320227 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:03:11.320920 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:03:11.320922 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:03:11.321244 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:03:11.321245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:03:11.321290 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:11.322252 (+ 962us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:03:11.322256 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:03:11.323389 (+ 1133us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:03:11.326044 (+ 2655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:03:11.326045 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:03:11.326046 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:03:11.326187 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:03:11.326366 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:03:11.326367 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:03:11.326367 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:03:11.326412 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:03:11.326654 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:03:11.326655 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:03:11.326689 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 05:03:11.326748 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 05:03:11.326842 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260924 05:03:11.326994 22821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:11.319300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34420 (local address 127.0.98.60:40015)
0924 05:03:11.319327 (+ 27us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:11.319328 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:11.319347 (+ 19us) server_negotiation.cc:407] Connection header received
0924 05:03:11.319400 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:11.319400 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:11.319408 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:11.319450 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:03:11.319672 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:11.320161 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:11.320988 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:11.321192 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:11.321241 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:11.323564 (+ 2323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:03:11.323565 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:03:11.323565 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:03:11.323571 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:03:11.325979 (+ 2408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:11.326241 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:11.326242 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:11.326242 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:11.326286 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:11.326459 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:11.326460 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:11.326460 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:11.326599 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:03:11.326652 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 05:03:11.326782 (+ 130us) server_negotiation.cc:299] Negotiation successful
0924 05:03:11.326889 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":14}
I20260924 05:03:11.328353 22816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80ca68509f87499d87566546a3372683: Bootstrap starting.
I20260924 05:03:11.328984 22816 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 80ca68509f87499d87566546a3372683: Neither blocks nor log segments found. Creating new log.
I20260924 05:03:11.329272 22816 log.cc:824] T 00000000000000000000000000000000 P 80ca68509f87499d87566546a3372683: Log is configured to *not* fsync() on all Append() calls
Sep 24 05:03:11 dist-test-slave-70st krb5kdc[22585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226190, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 05:03:11.330117 22816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80ca68509f87499d87566546a3372683: No bootstrap required, opened a new log
I20260924 05:03:11.331478 22816 raft_consensus.cc:348] T 00000000000000000000000000000000 P 80ca68509f87499d87566546a3372683 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44b689b40d724231be89dd39bbf456cf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44577 } } peers { permanent_uuid: "b7fb492c0e5f49218d8aa48b49192e48" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40007 } } peers { permanent_uuid: "80ca68509f87499d87566546a3372683" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40015 } }
I20260924 05:03:11.331598 22816 raft_consensus.cc:729] T 00000000000000000000000000000000 P 80ca68509f87499d87566546a3372683 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80ca68509f87499d87566546a3372683, State: Initialized, Role: FOLLOWER
I20260924 05:03:11.331676 22816 consensus_queue.cc:260] T 00000000000000000000000000000000 P 80ca68509f87499d87566546a3372683 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44b689b40d724231be89dd39bbf456cf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44577 } } peers { permanent_uuid: "b7fb492c0e5f49218d8aa48b49192e48" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40007 } } peers { permanent_uuid: "80ca68509f87499d87566546a3372683" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40015 } }
I20260924 05:03:11.331910 22837 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80ca68509f87499d87566546a3372683 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44b689b40d724231be89dd39bbf456cf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44577 } } peers { permanent_uuid: "b7fb492c0e5f49218d8aa48b49192e48" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40007 } } peers { permanent_uuid: "80ca68509f87499d87566546a3372683" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40015 } } }
I20260924 05:03:11.332027 22837 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80ca68509f87499d87566546a3372683 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 05:03:11.331970 22816 sys_catalog.cc:564] T 00000000000000000000000000000000 P 80ca68509f87499d87566546a3372683 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:03:11.334046 22746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:11.324021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45924 (local address 127.0.98.61:40007)
0924 05:03:11.324068 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:11.324069 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:11.324077 (+ 8us) server_negotiation.cc:407] Connection header received
0924 05:03:11.324162 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:11.324163 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:11.324172 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:11.324198 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:03:11.324773 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:11.325445 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:11.326380 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:11.326646 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:11.326683 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:11.329774 (+ 3091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:03:11.329775 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:03:11.329776 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:03:11.329782 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:03:11.332513 (+ 2731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:11.332867 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:11.332868 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:11.332868 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:11.332940 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:11.333192 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:11.333193 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:11.333193 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:11.333321 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:03:11.333831 (+ 510us) server_negotiation.cc:1035] Waiting for connection context
0924 05:03:11.333870 (+ 39us) server_negotiation.cc:299] Negotiation successful
0924 05:03:11.333964 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
W20260924 05:03:11.334101 22848 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 80ca68509f87499d87566546a3372683: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 05:03:11.334226 22848 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 05:03:11.335574 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 05:03:11.333869 22748 master_runner.cc:186] Error getting master registration for 127.0.98.62:44577: OK, instance_id { permanent_uuid: "44b689b40d724231be89dd39bbf456cf" instance_seqno: 1790226191008795 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 05:03:11.336067 22748 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 05:03:11.336534 22640 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "44b689b40d724231be89dd39bbf456cf"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37386
I20260924 05:03:11.354617 22666 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40015 } attempt: 2
Sep 24 05:03:11 dist-test-slave-70st krb5kdc[22585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226190, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 05:03:11.364405 22669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:11.354964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40015 (local address 127.0.0.1:34434)
0924 05:03:11.355029 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0924 05:03:11.355040 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 05:03:11.355081 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:03:11.355228 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:03:11.355228 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:03:11.355411 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:03:11.355463 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:03:11.355465 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:03:11.356345 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:03:11.356346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:03:11.357156 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:03:11.357158 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:03:11.357659 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:03:11.357660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:03:11.357723 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:11.358943 (+ 1220us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:03:11.358949 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:03:11.360401 (+ 1452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:03:11.363321 (+ 2920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:03:11.363322 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:03:11.363323 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:03:11.363539 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:03:11.363718 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:03:11.363718 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:03:11.363719 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:03:11.363794 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:03:11.364098 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:03:11.364099 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:03:11.364144 (+ 45us) client_negotiation.cc:769] Sending connection context
0924 05:03:11.364203 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 05:03:11.364299 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260924 05:03:11.364787 22821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:11.355054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34434 (local address 127.0.98.60:40015)
0924 05:03:11.355095 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:11.355097 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:11.355106 (+ 9us) server_negotiation.cc:407] Connection header received
0924 05:03:11.355132 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:11.355133 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:11.355143 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:11.355216 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:03:11.355538 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:11.356270 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:11.357249 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:11.357577 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:11.357635 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:11.360489 (+ 2854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:03:11.360490 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:03:11.360491 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:03:11.360498 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:03:11.363246 (+ 2748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:11.363611 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:11.363612 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:11.363613 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:11.363667 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:11.363875 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:11.363877 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:11.363877 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:11.364018 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:03:11.364074 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 05:03:11.364615 (+ 541us) server_negotiation.cc:299] Negotiation successful
0924 05:03:11.364694 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 05:03:11.366494 22666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44b689b40d724231be89dd39bbf456cf: Bootstrap starting.
I20260924 05:03:11.367218 22666 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 44b689b40d724231be89dd39bbf456cf: Neither blocks nor log segments found. Creating new log.
I20260924 05:03:11.367507 22666 log.cc:824] T 00000000000000000000000000000000 P 44b689b40d724231be89dd39bbf456cf: Log is configured to *not* fsync() on all Append() calls
I20260924 05:03:11.368378 22666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44b689b40d724231be89dd39bbf456cf: No bootstrap required, opened a new log
I20260924 05:03:11.369823 22666 raft_consensus.cc:348] T 00000000000000000000000000000000 P 44b689b40d724231be89dd39bbf456cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44b689b40d724231be89dd39bbf456cf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44577 } } peers { permanent_uuid: "b7fb492c0e5f49218d8aa48b49192e48" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40007 } } peers { permanent_uuid: "80ca68509f87499d87566546a3372683" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40015 } }
I20260924 05:03:11.370015 22666 raft_consensus.cc:729] T 00000000000000000000000000000000 P 44b689b40d724231be89dd39bbf456cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44b689b40d724231be89dd39bbf456cf, State: Initialized, Role: FOLLOWER
I20260924 05:03:11.370126 22666 consensus_queue.cc:260] T 00000000000000000000000000000000 P 44b689b40d724231be89dd39bbf456cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44b689b40d724231be89dd39bbf456cf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44577 } } peers { permanent_uuid: "b7fb492c0e5f49218d8aa48b49192e48" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40007 } } peers { permanent_uuid: "80ca68509f87499d87566546a3372683" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40015 } }
I20260924 05:03:11.370456 22666 sys_catalog.cc:564] T 00000000000000000000000000000000 P 44b689b40d724231be89dd39bbf456cf [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:03:11.370515 22640 raft_consensus.cc:482] T 00000000000000000000000000000000 P 44b689b40d724231be89dd39bbf456cf [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 05:03:11.370594 22640 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 44b689b40d724231be89dd39bbf456cf [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:03:11.371330 22640 raft_consensus.cc:504] T 00000000000000000000000000000000 P 44b689b40d724231be89dd39bbf456cf [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44b689b40d724231be89dd39bbf456cf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44577 } } peers { permanent_uuid: "b7fb492c0e5f49218d8aa48b49192e48" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40007 } } peers { permanent_uuid: "80ca68509f87499d87566546a3372683" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40015 } }
I20260924 05:03:11.371567 22640 leader_election.cc:288] T 00000000000000000000000000000000 P 44b689b40d724231be89dd39bbf456cf [CANDIDATE]: Term 1 election: Requested vote from peers b7fb492c0e5f49218d8aa48b49192e48 (127.0.98.61:40007), 80ca68509f87499d87566546a3372683 (127.0.98.60:40015)
I20260924 05:03:11.371986 22715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "44b689b40d724231be89dd39bbf456cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b7fb492c0e5f49218d8aa48b49192e48"
I20260924 05:03:11.372117 22715 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b7fb492c0e5f49218d8aa48b49192e48 [term 0 FOLLOWER]: Advancing to term 1
W20260924 05:03:11.372457 22862 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 44b689b40d724231be89dd39bbf456cf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 05:03:11.372501 22862 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 05:03:11.372977 22715 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b7fb492c0e5f49218d8aa48b49192e48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44b689b40d724231be89dd39bbf456cf in term 1.
I20260924 05:03:11.373215 22617 leader_election.cc:302] T 00000000000000000000000000000000 P 44b689b40d724231be89dd39bbf456cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44b689b40d724231be89dd39bbf456cf, b7fb492c0e5f49218d8aa48b49192e48; no voters:
I20260924 05:03:11.373901 22790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "44b689b40d724231be89dd39bbf456cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "80ca68509f87499d87566546a3372683"
I20260924 05:03:11.374027 22790 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 80ca68509f87499d87566546a3372683 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:03:11.374854 22790 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 80ca68509f87499d87566546a3372683 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44b689b40d724231be89dd39bbf456cf in term 1.
I20260924 05:03:11.378039 22598 master_runner.cc:263] No leader master found from master 44b689b40d724231be89dd39bbf456cf
I20260924 05:03:11.378084 22598 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 05:03:11.378576 22864 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44b689b40d724231be89dd39bbf456cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44b689b40d724231be89dd39bbf456cf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44577 } } peers { permanent_uuid: "b7fb492c0e5f49218d8aa48b49192e48" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40007 } } peers { permanent_uuid: "80ca68509f87499d87566546a3372683" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40015 } } }
I20260924 05:03:11.378758 22864 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44b689b40d724231be89dd39bbf456cf [sys.catalog]: This master's current role is: FOLLOWER
I20260924 05:03:11.378659 22851 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 44b689b40d724231be89dd39bbf456cf [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:03:11.385622 22851 raft_consensus.cc:686] T 00000000000000000000000000000000 P 44b689b40d724231be89dd39bbf456cf [term 1 LEADER]: Becoming Leader. State: Replica: 44b689b40d724231be89dd39bbf456cf, State: Running, Role: LEADER
I20260924 05:03:11.385897 22851 consensus_queue.cc:237] T 00000000000000000000000000000000 P 44b689b40d724231be89dd39bbf456cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44b689b40d724231be89dd39bbf456cf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44577 } } peers { permanent_uuid: "b7fb492c0e5f49218d8aa48b49192e48" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40007 } } peers { permanent_uuid: "80ca68509f87499d87566546a3372683" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40015 } }
I20260924 05:03:11.386711 22864 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44b689b40d724231be89dd39bbf456cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 44b689b40d724231be89dd39bbf456cf. Latest consensus state: current_term: 1 leader_uuid: "44b689b40d724231be89dd39bbf456cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44b689b40d724231be89dd39bbf456cf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44577 } } peers { permanent_uuid: "b7fb492c0e5f49218d8aa48b49192e48" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40007 } } peers { permanent_uuid: "80ca68509f87499d87566546a3372683" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40015 } } }
I20260924 05:03:11.386806 22864 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44b689b40d724231be89dd39bbf456cf [sys.catalog]: This master's current role is: LEADER
I20260924 05:03:11.387091 22873 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:03:11.387336 22873 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:03:11.390552 22670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:11.378981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37394 (local address 127.0.98.62:44577)
0924 05:03:11.379015 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:11.379016 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:11.379619 (+ 603us) server_negotiation.cc:407] Connection header received
0924 05:03:11.379701 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:11.379702 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:11.379711 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:11.379738 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:03:11.380408 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:11.381101 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:11.382240 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:11.382537 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:11.382573 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:11.384889 (+ 2316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:03:11.384890 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:03:11.384891 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:03:11.384898 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:03:11.389322 (+ 4424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:11.389718 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:11.389719 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:11.389720 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:11.389772 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:11.390001 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:11.390002 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:11.390002 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:11.390125 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:03:11.390189 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 05:03:11.390390 (+ 201us) server_negotiation.cc:299] Negotiation successful
0924 05:03:11.390463 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 05:03:11.390882 22746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:11.378368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45928 (local address 127.0.98.61:40007)
0924 05:03:11.378430 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:11.378432 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:11.379478 (+ 1046us) server_negotiation.cc:407] Connection header received
0924 05:03:11.379547 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:11.379548 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:11.379559 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:11.379623 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:03:11.380335 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:11.381030 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:11.382224 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:11.382523 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:11.382563 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:11.387051 (+ 4488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:03:11.387053 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:03:11.387053 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:03:11.387060 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:03:11.389745 (+ 2685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:11.390108 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:11.390109 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:11.390109 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:11.390160 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:11.390359 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:11.390360 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:11.390360 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:11.390476 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:03:11.390537 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 05:03:11.390718 (+ 181us) server_negotiation.cc:299] Negotiation successful
0924 05:03:11.390790 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 05:03:11.394222 22790 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 80ca68509f87499d87566546a3372683 [term 1 FOLLOWER]: Refusing update from remote peer 44b689b40d724231be89dd39bbf456cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:03:11.394539 22851 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 44b689b40d724231be89dd39bbf456cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "80ca68509f87499d87566546a3372683" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 05:03:11.395190 22694 tablet.cc:2369] T 00000000000000000000000000000000 P b7fb492c0e5f49218d8aa48b49192e48: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 05:03:11.395967 22837 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80ca68509f87499d87566546a3372683 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 44b689b40d724231be89dd39bbf456cf. Latest consensus state: current_term: 1 leader_uuid: "44b689b40d724231be89dd39bbf456cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44b689b40d724231be89dd39bbf456cf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44577 } } peers { permanent_uuid: "b7fb492c0e5f49218d8aa48b49192e48" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40007 } } peers { permanent_uuid: "80ca68509f87499d87566546a3372683" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40015 } } }
I20260924 05:03:11.396054 22837 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80ca68509f87499d87566546a3372683 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 05:03:11.396457 22715 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b7fb492c0e5f49218d8aa48b49192e48 [term 1 FOLLOWER]: Refusing update from remote peer 44b689b40d724231be89dd39bbf456cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:03:11.396726 22851 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 44b689b40d724231be89dd39bbf456cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7fb492c0e5f49218d8aa48b49192e48" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:03:11.397858 22823 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7fb492c0e5f49218d8aa48b49192e48 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 44b689b40d724231be89dd39bbf456cf. Latest consensus state: current_term: 1 leader_uuid: "44b689b40d724231be89dd39bbf456cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44b689b40d724231be89dd39bbf456cf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44577 } } peers { permanent_uuid: "b7fb492c0e5f49218d8aa48b49192e48" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40007 } } peers { permanent_uuid: "80ca68509f87499d87566546a3372683" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40015 } } }
I20260924 05:03:11.397948 22823 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7fb492c0e5f49218d8aa48b49192e48 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 05:03:11.398248 22821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:11.379241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34448 (local address 127.0.98.60:40015)
0924 05:03:11.379277 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:11.379279 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:11.388227 (+ 8948us) server_negotiation.cc:407] Connection header received
0924 05:03:11.388334 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:11.388335 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:11.388347 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:11.388375 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:03:11.389514 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:11.390228 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:11.391139 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:11.391396 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:11.391439 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:11.393539 (+ 2100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:03:11.393541 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:03:11.393541 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:03:11.393548 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:03:11.396599 (+ 3051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:11.396959 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:11.396961 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:11.396961 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:11.397012 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:11.397353 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:11.397354 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:11.397355 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:11.397545 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:03:11.397624 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 05:03:11.398082 (+ 458us) server_negotiation.cc:299] Negotiation successful
0924 05:03:11.398159 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 05:03:11.402509 22851 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44b689b40d724231be89dd39bbf456cf [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "44b689b40d724231be89dd39bbf456cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44b689b40d724231be89dd39bbf456cf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44577 } } peers { permanent_uuid: "b7fb492c0e5f49218d8aa48b49192e48" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40007 } } peers { permanent_uuid: "80ca68509f87499d87566546a3372683" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40015 } } }
I20260924 05:03:11.402597 22851 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44b689b40d724231be89dd39bbf456cf [sys.catalog]: This master's current role is: LEADER
I20260924 05:03:11.402632 22851 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44b689b40d724231be89dd39bbf456cf [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "44b689b40d724231be89dd39bbf456cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44b689b40d724231be89dd39bbf456cf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44577 } } peers { permanent_uuid: "b7fb492c0e5f49218d8aa48b49192e48" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40007 } } peers { permanent_uuid: "80ca68509f87499d87566546a3372683" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40015 } } }
I20260924 05:03:11.402666 22851 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44b689b40d724231be89dd39bbf456cf [sys.catalog]: This master's current role is: LEADER
I20260924 05:03:11.403185 22823 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7fb492c0e5f49218d8aa48b49192e48 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "44b689b40d724231be89dd39bbf456cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44b689b40d724231be89dd39bbf456cf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44577 } } peers { permanent_uuid: "b7fb492c0e5f49218d8aa48b49192e48" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40007 } } peers { permanent_uuid: "80ca68509f87499d87566546a3372683" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40015 } } }
I20260924 05:03:11.403298 22823 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7fb492c0e5f49218d8aa48b49192e48 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 05:03:11.403460 22877 mvcc.cc:205] Tried to move back new op lower bound from 7332766479948136448 to 7332766479918018560. Current Snapshot: MvccSnapshot[applied={T|T < 7332766479948136448}]
I20260924 05:03:11.403808 22837 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80ca68509f87499d87566546a3372683 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "44b689b40d724231be89dd39bbf456cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44b689b40d724231be89dd39bbf456cf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44577 } } peers { permanent_uuid: "b7fb492c0e5f49218d8aa48b49192e48" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40007 } } peers { permanent_uuid: "80ca68509f87499d87566546a3372683" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40015 } } }
I20260924 05:03:11.403883 22837 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80ca68509f87499d87566546a3372683 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 05:03:11.404506 22874 mvcc.cc:205] Tried to move back new op lower bound from 7332766479948136448 to 7332766479918018560. Current Snapshot: MvccSnapshot[applied={T|T < 7332766479948136448}]
I20260924 05:03:11.405004 22875 mvcc.cc:205] Tried to move back new op lower bound from 7332766479948136448 to 7332766479918018560. Current Snapshot: MvccSnapshot[applied={T|T < 7332766479948136448}]
I20260924 05:03:11.405588 22873 catalog_manager.cc:1347] Generated new cluster ID: c4ee754ae45944dfab78f361ec4e35be
I20260924 05:03:11.405642 22873 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:03:11.418835 22873 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:03:11.419611 22873 catalog_manager.cc:1504] Loading token signing keys...
I20260924 05:03:11.427223 22873 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 44b689b40d724231be89dd39bbf456cf: Generated new TSK 0
I20260924 05:03:11.427490 22873 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 05:03:11.459704 22670 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:03:11.457766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37400 (local address 127.0.98.62:44577)
0924 05:03:11.457812 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:11.457814 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:11.458759 (+ 945us) server_negotiation.cc:407] Connection header received
0924 05:03:11.458859 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:11.458860 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:11.458872 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:11.458899 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:03:11.459632 (+ 733us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37400: BlockingRecv error: recv got EOF from 127.0.0.1:37400 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260924 05:03:11.460621 22746 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:03:11.458029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45942 (local address 127.0.98.61:40007)
0924 05:03:11.458070 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:11.458071 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:11.460062 (+ 1991us) server_negotiation.cc:407] Connection header received
0924 05:03:11.460160 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:11.460161 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:11.460172 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:11.460200 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:03:11.460555 (+ 355us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45942: BlockingRecv error: recv got EOF from 127.0.0.1:45942 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260924 05:03:11.461519 22821 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:03:11.458403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34450 (local address 127.0.98.60:40015)
0924 05:03:11.458443 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:11.458445 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:11.461000 (+ 2555us) server_negotiation.cc:407] Connection header received
0924 05:03:11.461090 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:11.461091 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:11.461102 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:11.461129 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:03:11.461458 (+ 329us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34450: BlockingRecv error: recv got EOF from 127.0.0.1:34450 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260924 05:03:11.464066 22746 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:03:11.463333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45948 (local address 127.0.98.61:40007)
0924 05:03:11.463370 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:11.463372 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:11.463548 (+ 176us) server_negotiation.cc:407] Connection header received
0924 05:03:11.463647 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:11.463648 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:11.463666 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:11.463693 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:03:11.464020 (+ 327us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45948: BlockingRecv error: recv got EOF from 127.0.0.1:45948 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260924 05:03:11.467185 22746 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:03:11.466397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45960 (local address 127.0.98.61:40007)
0924 05:03:11.466432 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:11.466434 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:11.466668 (+ 234us) server_negotiation.cc:407] Connection header received
0924 05:03:11.466779 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:11.466780 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:11.466790 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:11.466819 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:03:11.467136 (+ 317us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45960: BlockingRecv error: recv got EOF from 127.0.0.1:45960 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260924 05:03:11.471082 22746 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:03:11.470305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45972 (local address 127.0.98.61:40007)
0924 05:03:11.470339 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:11.470340 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:11.470558 (+ 218us) server_negotiation.cc:407] Connection header received
0924 05:03:11.470698 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:11.470699 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:11.470709 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:11.470746 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:03:11.471039 (+ 293us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45972: BlockingRecv error: recv got EOF from 127.0.0.1:45972 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260924 05:03:11.477106 22746 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:03:11.476266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45982 (local address 127.0.98.61:40007)
0924 05:03:11.476301 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:11.476302 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:11.476568 (+ 266us) server_negotiation.cc:407] Connection header received
0924 05:03:11.476664 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:11.476665 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:11.476676 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:11.476705 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:03:11.477054 (+ 349us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45982: BlockingRecv error: recv got EOF from 127.0.0.1:45982 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260924 05:03:11.487586 22746 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:03:11.486629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45990 (local address 127.0.98.61:40007)
0924 05:03:11.486674 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:11.486676 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:11.486989 (+ 313us) server_negotiation.cc:407] Connection header received
0924 05:03:11.487094 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:11.487096 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:11.487108 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:11.487138 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:03:11.487525 (+ 387us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45990: BlockingRecv error: recv got EOF from 127.0.0.1:45990 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260924 05:03:11.506193 22746 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:03:11.505156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46002 (local address 127.0.98.61:40007)
0924 05:03:11.505207 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:11.505209 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:11.505572 (+ 363us) server_negotiation.cc:407] Connection header received
0924 05:03:11.505677 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:11.505678 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:11.505691 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:11.505720 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:03:11.506127 (+ 407us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46002: BlockingRecv error: recv got EOF from 127.0.0.1:46002 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260924 05:03:11.540037 22746 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:03:11.539427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46004 (local address 127.0.98.61:40007)
0924 05:03:11.539484 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:11.539487 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:11.539566 (+ 79us) server_negotiation.cc:407] Connection header received
0924 05:03:11.539613 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:11.539614 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:11.539624 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:11.539682 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:03:11.539982 (+ 300us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46004: BlockingRecv error: recv got EOF from 127.0.0.1:46004 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
W20260924 05:03:11.605868 22746 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:03:11.605154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46008 (local address 127.0.98.61:40007)
0924 05:03:11.605229 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:11.605232 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:11.605305 (+ 73us) server_negotiation.cc:407] Connection header received
0924 05:03:11.605381 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:11.605382 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:11.605394 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:11.605466 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:03:11.605803 (+ 337us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46008: BlockingRecv error: recv got EOF from 127.0.0.1:46008 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260924 05:03:11.735534 22746 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:03:11.734855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46014 (local address 127.0.98.61:40007)
0924 05:03:11.734919 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:11.734922 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:11.735020 (+ 98us) server_negotiation.cc:407] Connection header received
0924 05:03:11.735095 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:11.735096 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:11.735108 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:11.735161 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:03:11.735472 (+ 311us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46014: BlockingRecv error: recv got EOF from 127.0.0.1:46014 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
W20260924 05:03:11.993206 22746 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:03:11.992631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46022 (local address 127.0.98.61:40007)
0924 05:03:11.992685 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:11.992687 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:11.992696 (+ 9us) server_negotiation.cc:407] Connection header received
0924 05:03:11.992757 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:11.992758 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:11.992769 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:11.992815 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:03:11.993144 (+ 329us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46022: BlockingRecv error: recv got EOF from 127.0.0.1:46022 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 05:03:12.316013 22834 catalog_manager.cc:1259] Loaded cluster ID: c4ee754ae45944dfab78f361ec4e35be
I20260924 05:03:12.316085 22834 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b7fb492c0e5f49218d8aa48b49192e48: loading cluster ID for follower catalog manager: success
I20260924 05:03:12.316915 22834 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b7fb492c0e5f49218d8aa48b49192e48: acquiring CA information for follower catalog manager: success
I20260924 05:03:12.317224 22834 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b7fb492c0e5f49218d8aa48b49192e48: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 05:03:12.334647 22848 catalog_manager.cc:1259] Loaded cluster ID: c4ee754ae45944dfab78f361ec4e35be
I20260924 05:03:12.334720 22848 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 80ca68509f87499d87566546a3372683: loading cluster ID for follower catalog manager: success
I20260924 05:03:12.335543 22848 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 80ca68509f87499d87566546a3372683: acquiring CA information for follower catalog manager: success
I20260924 05:03:12.335803 22848 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 80ca68509f87499d87566546a3372683: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 05:03:12.508863 22947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:12.506343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46026 (local address 127.0.98.61:40007)
0924 05:03:12.506454 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:12.506457 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:12.506549 (+ 92us) server_negotiation.cc:407] Connection header received
0924 05:03:12.506626 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:12.506628 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:12.506681 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:12.506750 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:03:12.506886 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:12.507386 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:12.508127 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:12.508319 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:12.508373 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:12.508531 (+ 158us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:03:12.508641 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:03:12.508709 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 05:03:12.508754 (+ 45us) server_negotiation.cc:299] Negotiation successful
0924 05:03:12.508774 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":53,"threads_started":1}
I20260924 05:03:12.512671 22953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:12.510499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34462 (local address 127.0.98.60:40015)
0924 05:03:12.510609 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:12.510611 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:12.510659 (+ 48us) server_negotiation.cc:407] Connection header received
0924 05:03:12.510713 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:12.510714 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:12.510748 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:12.510816 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:03:12.510891 (+ 75us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:12.511366 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:12.512065 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:12.512262 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:12.512314 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:12.512409 (+ 95us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:03:12.512510 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:03:12.512561 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 05:03:12.512577 (+ 16us) server_negotiation.cc:299] Negotiation successful
0924 05:03:12.512592 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":53,"threads_started":1}
I20260924 05:03:12.517100 22960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:12.514296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37404 (local address 127.0.98.62:44577)
0924 05:03:12.514407 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:12.514409 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:12.514492 (+ 83us) server_negotiation.cc:407] Connection header received
0924 05:03:12.514539 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:12.514540 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:12.514590 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:12.514670 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:03:12.514846 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:12.515551 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:12.516238 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:12.516516 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:12.516601 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:12.516780 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:03:12.516892 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:03:12.516965 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 05:03:12.516984 (+ 19us) server_negotiation.cc:299] Negotiation successful
0924 05:03:12.517009 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":57,"threads_started":1}
I20260924 05:03:12.517174 22947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:12.514254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46038 (local address 127.0.98.61:40007)
0924 05:03:12.514297 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:12.514299 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:12.514426 (+ 127us) server_negotiation.cc:407] Connection header received
0924 05:03:12.514463 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:12.514464 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:12.514475 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:12.514519 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:03:12.514600 (+ 81us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:12.515328 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:12.516358 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:12.516654 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:12.516705 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:12.516824 (+ 119us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:03:12.516857 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:03:12.516897 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0924 05:03:12.517101 (+ 204us) server_negotiation.cc:299] Negotiation successful
0924 05:03:12.517117 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 05:03:12.517259 22953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:12.514421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34470 (local address 127.0.98.60:40015)
0924 05:03:12.514476 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:12.514478 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:12.514677 (+ 199us) server_negotiation.cc:407] Connection header received
0924 05:03:12.514748 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:12.514750 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:12.514759 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:12.514824 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:03:12.514988 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:12.515631 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:12.516545 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:12.516799 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:12.516851 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:12.517027 (+ 176us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:03:12.517055 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:03:12.517079 (+ 24us) server_negotiation.cc:1035] Waiting for connection context
0924 05:03:12.517173 (+ 94us) server_negotiation.cc:299] Negotiation successful
0924 05:03:12.517194 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 05:03:12.528949 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 22598
I20260924 05:03:12.533536 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 22673
I20260924 05:03:12.537669 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 22748
2026-09-24T05:03:12Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2702 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225929848825-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 05:03:12 dist-test-slave-70st krb5kdc[22964](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 05:03:12 dist-test-slave-70st krb5kdc[22964](info): set up 2 sockets
Sep 24 05:03:12 dist-test-slave-70st krb5kdc[22964](info): commencing operation
krb5kdc: starting...
W20260924 05:03:13.481287 392 mini_kdc.cc:121] Time spent starting KDC: real 0.926s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225929848825-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225929848825-392-0/krb5kdc/test-user.keytab.
Sep 24 05:03:13 dist-test-slave-70st krb5kdc[22964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226193, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T05:03:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T05:03:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 05:03:13.594076 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225929848825-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225929848825-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225929848825-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36551
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36551,127.0.98.61:38845,127.0.98.60:35017
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 05:03:13.672358 22977 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 05:03:13.672574 22977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:03:13.672598 22977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:03:13.672623 22977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:03:13.673760 22977 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:03:13.673797 22977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:03:13.673815 22977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:03:13.673830 22977 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:03:13.673846 22977 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:03:13.675294 22977 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225929848825-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225929848825-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36551,127.0.98.61:38845,127.0.98.60:35017
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225929848825-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36551
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22977
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225929848825-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:03:13.675535 22977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:03:13.675733 22977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 05:03:13.680433 22977 server_base.cc:1034] running on GCE node
W20260924 05:03:13.680490 22983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:03:13.680388 22982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:03:13.680399 22985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:03:13.680912 22977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:03:13.681171 22977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:03:13.682351 22977 hybrid_clock.cc:648] HybridClock initialized: now 1790226193682332 us; error 30 us; skew 500 ppm
Sep 24 05:03:13 dist-test-slave-70st krb5kdc[22964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226193, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:03:13.685161 22977 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 05:03:13.685675 22977 webserver.cc:460] Webserver started at http://127.0.98.62:34397/ using document root <none> and password file <none>
I20260924 05:03:13.685928 22977 fs_manager.cc:359] Metadata directory not provided
I20260924 05:03:13.685992 22977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:03:13.686158 22977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:03:13.687376 22977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225929848825-392-0/minicluster-data/master-0/data/instance:
uuid: "781e6faed8424949b8f05fddf330a60b"
format_stamp: "Formatted at 2026-09-24 05:03:13 on dist-test-slave-70st"
server_key: "f4f4edc225c22c993c5f732f6afb529b"
server_key_iv: "b64ceeeb46ec36ff1eb0bec47ad38e24"
server_key_version: "encryptionkey@0"
I20260924 05:03:13.687659 22977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225929848825-392-0/minicluster-data/master-0/wal/instance:
uuid: "781e6faed8424949b8f05fddf330a60b"
format_stamp: "Formatted at 2026-09-24 05:03:13 on dist-test-slave-70st"
server_key: "f4f4edc225c22c993c5f732f6afb529b"
server_key_iv: "b64ceeeb46ec36ff1eb0bec47ad38e24"
server_key_version: "encryptionkey@0"
I20260924 05:03:13.688840 22977 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 05:03:13.689447 22992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:03:13.689603 22977 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:03:13.689683 22977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225929848825-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225929848825-392-0/minicluster-data/master-0/wal
uuid: "781e6faed8424949b8f05fddf330a60b"
format_stamp: "Formatted at 2026-09-24 05:03:13 on dist-test-slave-70st"
server_key: "f4f4edc225c22c993c5f732f6afb529b"
server_key_iv: "b64ceeeb46ec36ff1eb0bec47ad38e24"
server_key_version: "encryptionkey@0"
I20260924 05:03:13.689751 22977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225929848825-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225929848825-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225929848825-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:03:13.703764 22977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:03:13.705902 22977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:03:13.706035 22977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:03:13.709738 22977 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36551
I20260924 05:03:13.709774 23044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36551 every 8 connection(s)
I20260924 05:03:13.710105 22977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225929848825-392-0/minicluster-data/master-0/data/info.pb
I20260924 05:03:13.711023 23045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:03:13.712505 23045 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36551 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 05:03:13 dist-test-slave-70st krb5kdc[22964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226193, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:03:13.718732 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 22977
I20260924 05:03:13.718834 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225929848825-392-0/minicluster-data/master-0/wal/instance
I20260924 05:03:13.719035 392 external_mini_cluster.cc:1428] Setting key dedec7e80fe806b31675590540d178b1
I20260924 05:03:13.721830 23048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:13.713318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59026 (local address 127.0.98.62:36551)
0924 05:03:13.713470 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:13.713475 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:13.713507 (+ 32us) server_negotiation.cc:407] Connection header received
0924 05:03:13.713962 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:13.713964 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:13.713982 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:13.714067 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:03:13.714603 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:13.715452 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:13.716296 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:13.716603 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:13.716690 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:13.718651 (+ 1961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:03:13.718660 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:03:13.718660 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:03:13.718668 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:03:13.720295 (+ 1627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:13.720730 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:13.720732 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:13.720733 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:13.720825 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:13.721030 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:13.721031 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:13.721031 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:13.721234 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:03:13.721301 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 05:03:13.721519 (+ 218us) server_negotiation.cc:299] Negotiation successful
0924 05:03:13.721641 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":14,"server-negotiator.queue_time_us":121,"thread_start_us":50,"threads_started":1}
I20260924 05:03:13.721836 23047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:13.713126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36551 (local address 127.0.0.1:59026)
0924 05:03:13.713286 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 05:03:13.713320 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 05:03:13.713807 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:03:13.714080 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:03:13.714083 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:03:13.714324 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:03:13.714496 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:03:13.714500 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:03:13.715528 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:03:13.715528 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:03:13.716189 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:03:13.716191 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:03:13.716679 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:03:13.716680 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:03:13.716729 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:13.717210 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:03:13.717218 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:03:13.718576 (+ 1358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:03:13.720368 (+ 1792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:03:13.720375 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:03:13.720378 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:03:13.720648 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:03:13.720878 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:03:13.720879 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:03:13.720879 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:03:13.720976 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:03:13.721292 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:03:13.721297 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:03:13.721371 (+ 74us) client_negotiation.cc:769] Sending connection context
0924 05:03:13.721499 (+ 128us) client_negotiation.cc:240] Negotiation successful
0924 05:03:13.721713 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"mutex_wait_us":65,"thread_start_us":87,"threads_started":1}
I20260924 05:03:13.722537 23045 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38845 } has no permanent_uuid. Determining permanent_uuid...
I20260924 05:03:13.722888 23047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:13.722772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38845 (local address 127.0.0.1:59208)
0924 05:03:13.722819 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 05:03:13.722866 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260924 05:03:13.723052 22994 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38845: connect: Connection refused (error 111)
I20260924 05:03:13.723505 23047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:13.723399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38845 (local address 127.0.0.1:59210)
0924 05:03:13.723452 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0924 05:03:13.723482 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260924 05:03:13.723601 23045 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:38845: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38845: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225929848825-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225929848825-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260924 05:03:13.757499 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225929848825-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225929848825-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225929848825-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225929848825-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:38845
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225929848825-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36551,127.0.98.61:38845,127.0.98.60:35017
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 05:03:13.772840 23045 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38845 } attempt: 1
I20260924 05:03:13.773341 23047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:13.773183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38845 (local address 127.0.0.1:59222)
0924 05:03:13.773245 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0924 05:03:13.773308 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260924 05:03:13.773689 23047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:13.773607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38845 (local address 127.0.0.1:59224)
0924 05:03:13.773643 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0924 05:03:13.773668 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260924 05:03:13.773775 23045 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:38845: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38845: connect: Connection refused (error 111)
W20260924 05:03:13.836006 23052 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 05:03:13.836228 23052 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:03:13.836261 23052 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:03:13.836287 23052 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:03:13.837411 23052 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:03:13.837471 23052 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:03:13.837488 23052 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:03:13.837504 23052 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:03:13.837519 23052 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:03:13.838974 23052 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225929848825-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225929848825-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36551,127.0.98.61:38845,127.0.98.60:35017
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225929848825-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:38845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225929848825-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23052
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225929848825-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:03:13.839206 23052 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:03:13.839413 23052 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 05:03:13.841938 23045 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38845 } attempt: 2
I20260924 05:03:13.842378 23047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:13.842228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38845 (local address 127.0.0.1:59236)
0924 05:03:13.842312 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0924 05:03:13.842347 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260924 05:03:13.842789 23047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:13.842660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38845 (local address 127.0.0.1:59250)
0924 05:03:13.842724 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0924 05:03:13.842760 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260924 05:03:13.842965 23045 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:38845: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38845: connect: Connection refused (error 111)
W20260924 05:03:13.844110 23060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:03:13.844149 23058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:03:13.844235 23052 server_base.cc:1034] running on GCE node
W20260924 05:03:13.844344 23057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:03:13.844578 23052 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:03:13.844818 23052 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:03:13.845938 23052 hybrid_clock.cc:648] HybridClock initialized: now 1790226193845932 us; error 29 us; skew 500 ppm
Sep 24 05:03:13 dist-test-slave-70st krb5kdc[22964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226193, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:03:13.848237 23052 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260924 05:03:13.848675 23052 webserver.cc:460] Webserver started at http://127.0.98.61:39907/ using document root <none> and password file <none>
I20260924 05:03:13.848862 23052 fs_manager.cc:359] Metadata directory not provided
I20260924 05:03:13.848908 23052 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:03:13.849027 23052 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:03:13.850095 23052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225929848825-392-0/minicluster-data/master-1/data/instance:
uuid: "e976476875d345c7ac7958650a004c56"
format_stamp: "Formatted at 2026-09-24 05:03:13 on dist-test-slave-70st"
server_key: "67d2cc08aa8dc2fd374f1273b9ea0651"
server_key_iv: "bd0019f7f33615532df9837f7559f397"
server_key_version: "encryptionkey@0"
I20260924 05:03:13.850412 23052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225929848825-392-0/minicluster-data/master-1/wal/instance:
uuid: "e976476875d345c7ac7958650a004c56"
format_stamp: "Formatted at 2026-09-24 05:03:13 on dist-test-slave-70st"
server_key: "67d2cc08aa8dc2fd374f1273b9ea0651"
server_key_iv: "bd0019f7f33615532df9837f7559f397"
server_key_version: "encryptionkey@0"
I20260924 05:03:13.851611 23052 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 05:03:13.852234 23067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:03:13.852407 23052 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 05:03:13.852491 23052 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225929848825-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225929848825-392-0/minicluster-data/master-1/wal
uuid: "e976476875d345c7ac7958650a004c56"
format_stamp: "Formatted at 2026-09-24 05:03:13 on dist-test-slave-70st"
server_key: "67d2cc08aa8dc2fd374f1273b9ea0651"
server_key_iv: "bd0019f7f33615532df9837f7559f397"
server_key_version: "encryptionkey@0"
I20260924 05:03:13.852546 23052 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225929848825-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225929848825-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225929848825-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:03:13.877349 23052 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:03:13.879160 23052 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:03:13.879288 23052 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:03:13.882938 23052 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:38845
I20260924 05:03:13.882970 23119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:38845 every 8 connection(s)
I20260924 05:03:13.883285 23052 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225929848825-392-0/minicluster-data/master-1/data/info.pb
I20260924 05:03:13.884176 23120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:03:13.885692 23120 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36551 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 05:03:13 dist-test-slave-70st krb5kdc[22964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226193, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:03:13.892159 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 23052
I20260924 05:03:13.892251 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225929848825-392-0/minicluster-data/master-1/wal/instance
I20260924 05:03:13.892413 392 external_mini_cluster.cc:1428] Setting key 4df8e62280a7e8d71d65385993c02c7b
I20260924 05:03:13.893792 23048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:13.886348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59042 (local address 127.0.98.62:36551)
0924 05:03:13.886401 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:13.886403 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:13.886593 (+ 190us) server_negotiation.cc:407] Connection header received
0924 05:03:13.887078 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:13.887079 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:13.887091 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:13.887142 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:03:13.887714 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:13.888192 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:13.889107 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:13.889307 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:13.889362 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:13.891311 (+ 1949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:03:13.891312 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:03:13.891313 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:03:13.891319 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:03:13.892404 (+ 1085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:13.892851 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:13.892852 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:13.892852 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:13.892904 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:13.893135 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:13.893136 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:13.893136 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:13.893231 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:03:13.893274 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 05:03:13.893605 (+ 331us) server_negotiation.cc:299] Negotiation successful
0924 05:03:13.893700 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 05:03:13.893957 23122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:13.886322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36551 (local address 127.0.0.1:59042)
0924 05:03:13.886481 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 05:03:13.886516 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 05:03:13.886981 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:03:13.887208 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:03:13.887212 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:03:13.887441 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:03:13.887629 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:03:13.887633 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:03:13.888267 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:03:13.888268 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:03:13.889031 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:03:13.889034 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:03:13.889389 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:03:13.889391 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:03:13.889527 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:13.889933 (+ 406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:03:13.889952 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:03:13.891225 (+ 1273us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:03:13.892491 (+ 1266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:03:13.892499 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:03:13.892500 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:03:13.892789 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:03:13.892955 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:03:13.892956 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:03:13.892956 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:03:13.893083 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:03:13.893277 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:03:13.893281 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:03:13.893450 (+ 169us) client_negotiation.cc:769] Sending connection context
0924 05:03:13.893565 (+ 115us) client_negotiation.cc:240] Negotiation successful
0924 05:03:13.893721 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":74,"threads_started":1}
I20260924 05:03:13.894625 23120 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38845 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 05:03:13 dist-test-slave-70st krb5kdc[22964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226193, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260924 05:03:13.901473 23122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:13.894843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38845 (local address 127.0.0.1:59254)
0924 05:03:13.894888 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 05:03:13.894898 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 05:03:13.894938 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:03:13.895155 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:03:13.895156 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:03:13.895286 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:03:13.895337 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:03:13.895338 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:03:13.896176 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:03:13.896177 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:03:13.896937 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:03:13.896950 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:03:13.897401 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:03:13.897402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:03:13.897491 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:13.897832 (+ 341us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:03:13.897837 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:03:13.899017 (+ 1180us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:03:13.900614 (+ 1597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:03:13.900615 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:03:13.900615 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:03:13.900760 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:03:13.900980 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:03:13.900981 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:03:13.900981 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:03:13.901024 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:03:13.901226 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:03:13.901227 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:03:13.901260 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 05:03:13.901300 (+ 40us) client_negotiation.cc:240] Negotiation successful
0924 05:03:13.901373 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260924 05:03:13.901592 23124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:13.894886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59254 (local address 127.0.98.61:38845)
0924 05:03:13.895034 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:13.895038 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:13.895055 (+ 17us) server_negotiation.cc:407] Connection header received
0924 05:03:13.895075 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:13.895077 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:13.895088 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:13.895145 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:03:13.895401 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:13.896104 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:13.897027 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:13.897333 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:13.897388 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:13.899089 (+ 1701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:03:13.899097 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:03:13.899097 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:03:13.899103 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:03:13.900542 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:13.900849 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:13.900852 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:13.900852 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:13.900920 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:13.901057 (+ 137us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:13.901058 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:13.901058 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:13.901186 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:03:13.901241 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 05:03:13.901332 (+ 91us) server_negotiation.cc:299] Negotiation successful
0924 05:03:13.901468 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":119,"thread_start_us":78,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260924 05:03:13.902168 23120 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35017 } has no permanent_uuid. Determining permanent_uuid...
I20260924 05:03:13.902513 23122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:13.902379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35017 (local address 127.0.0.1:51428)
0924 05:03:13.902427 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 05:03:13.902487 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260924 05:03:13.902644 23071 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35017: connect: Connection refused (error 111)
I20260924 05:03:13.903012 23122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:13.902933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35017 (local address 127.0.0.1:51440)
0924 05:03:13.902968 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0924 05:03:13.902993 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260924 05:03:13.903105 23120 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35017: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35017: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225929848825-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225929848825-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260924 05:03:13.915979 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskZbcc3B/build/release/bin/kudu
/tmp/dist-test-taskZbcc3B/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225929848825-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225929848825-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225929848825-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225929848825-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:35017
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225929848825-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36551,127.0.98.61:38845,127.0.98.60:35017
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225929848825-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225929848825-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225929848825-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 05:03:13.934109 23045 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38845 } attempt: 3
Sep 24 05:03:13 dist-test-slave-70st krb5kdc[22964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226193, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 05:03:13.940565 23124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:13.934555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59264 (local address 127.0.98.61:38845)
0924 05:03:13.934616 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:13.934619 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:13.934627 (+ 8us) server_negotiation.cc:407] Connection header received
0924 05:03:13.934653 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:13.934654 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:13.934664 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:13.934709 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:03:13.935013 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:13.935499 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:13.936171 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:13.936381 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:13.936436 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:13.938326 (+ 1890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:03:13.938327 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:03:13.938328 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:03:13.938335 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:03:13.939600 (+ 1265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:13.939885 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:13.939885 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:13.939886 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:13.939949 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:13.940104 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:13.940105 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:13.940106 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:13.940209 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:03:13.940261 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 05:03:13.940391 (+ 130us) server_negotiation.cc:299] Negotiation successful
0924 05:03:13.940463 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 05:03:13.940565 23047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:13.934473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38845 (local address 127.0.0.1:59264)
0924 05:03:13.934529 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0924 05:03:13.934540 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 05:03:13.934583 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:03:13.934719 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:03:13.934719 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:03:13.934868 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:03:13.934931 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:03:13.934933 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:03:13.935571 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:03:13.935572 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:03:13.936096 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:03:13.936098 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:03:13.936447 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:03:13.936448 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:03:13.936515 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:13.936913 (+ 398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:03:13.936918 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:03:13.938248 (+ 1330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:03:13.939671 (+ 1423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:03:13.939672 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:03:13.939672 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:03:13.939828 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:03:13.940001 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:03:13.940002 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:03:13.940002 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:03:13.940058 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:03:13.940260 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:03:13.940260 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:03:13.940306 (+ 46us) client_negotiation.cc:769] Sending connection context
0924 05:03:13.940364 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 05:03:13.940464 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260924 05:03:13.941105 23045 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35017 } has no permanent_uuid. Determining permanent_uuid...
I20260924 05:03:13.941485 23047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:13.941345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35017 (local address 127.0.0.1:51442)
0924 05:03:13.941385 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0924 05:03:13.941460 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260924 05:03:13.941771 23047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:13.941692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35017 (local address 127.0.0.1:51454)
0924 05:03:13.941727 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0924 05:03:13.941749 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260924 05:03:13.941847 23045 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35017: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35017: connect: Connection refused (error 111)
I20260924 05:03:13.952343 23120 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35017 } attempt: 1
I20260924 05:03:13.952836 23122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:13.952697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35017 (local address 127.0.0.1:51462)
0924 05:03:13.952758 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0924 05:03:13.952803 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260924 05:03:13.953118 23122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:13.953043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35017 (local address 127.0.0.1:51466)
0924 05:03:13.953078 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0924 05:03:13.953100 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260924 05:03:13.953205 23120 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35017: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35017: connect: Connection refused (error 111)
I20260924 05:03:13.972997 23045 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35017 } attempt: 1
I20260924 05:03:13.973511 23047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:13.973358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35017 (local address 127.0.0.1:51472)
0924 05:03:13.973438 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0924 05:03:13.973482 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260924 05:03:13.973839 23047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:13.973746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35017 (local address 127.0.0.1:51482)
0924 05:03:13.973788 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0924 05:03:13.973816 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260924 05:03:13.973932 23045 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35017: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35017: connect: Connection refused (error 111)
W20260924 05:03:13.996011 23127 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 05:03:13.996245 23127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 05:03:13.996263 23127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 05:03:13.996281 23127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 05:03:13.997664 23127 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 05:03:13.997722 23127 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 05:03:13.997736 23127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 05:03:13.997754 23127 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 05:03:13.997766 23127 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 05:03:13.999291 23127 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225929848825-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225929848825-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36551,127.0.98.61:38845,127.0.98.60:35017
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225929848825-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:35017
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225929848825-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23127
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225929848825-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 04:08:41 UTC on ccycloud.quasar-xrgdau.root.comops.site
build id 4669300
I20260924 05:03:13.999490 23127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 05:03:13.999676 23127 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 05:03:14.004321 23135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:03:14.004333 23133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 05:03:14.004475 23132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 05:03:14.004465 23127 server_base.cc:1034] running on GCE node
I20260924 05:03:14.004806 23127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 05:03:14.005062 23127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 05:03:14.006201 23127 hybrid_clock.cc:648] HybridClock initialized: now 1790226194006192 us; error 27 us; skew 500 ppm
Sep 24 05:03:14 dist-test-slave-70st krb5kdc[22964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226194, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 05:03:14.008513 23127 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260924 05:03:14.008929 23127 webserver.cc:460] Webserver started at http://127.0.98.60:44107/ using document root <none> and password file <none>
I20260924 05:03:14.009157 23127 fs_manager.cc:359] Metadata directory not provided
I20260924 05:03:14.009203 23127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 05:03:14.009289 23127 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 05:03:14.010357 23127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225929848825-392-0/minicluster-data/master-2/data/instance:
uuid: "8bfb729822d94bbc902c61c8d5938309"
format_stamp: "Formatted at 2026-09-24 05:03:14 on dist-test-slave-70st"
server_key: "4f2fe24720a4574ed2dbdc9e56f6999b"
server_key_iv: "6082a835521c13f47b69896f64588a01"
server_key_version: "encryptionkey@0"
I20260924 05:03:14.010643 23127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225929848825-392-0/minicluster-data/master-2/wal/instance:
uuid: "8bfb729822d94bbc902c61c8d5938309"
format_stamp: "Formatted at 2026-09-24 05:03:14 on dist-test-slave-70st"
server_key: "4f2fe24720a4574ed2dbdc9e56f6999b"
server_key_iv: "6082a835521c13f47b69896f64588a01"
server_key_version: "encryptionkey@0"
I20260924 05:03:14.011788 23127 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 05:03:14.012463 23142 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 05:03:14.012679 23127 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 05:03:14.012759 23127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225929848825-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225929848825-392-0/minicluster-data/master-2/wal
uuid: "8bfb729822d94bbc902c61c8d5938309"
format_stamp: "Formatted at 2026-09-24 05:03:14 on dist-test-slave-70st"
server_key: "4f2fe24720a4574ed2dbdc9e56f6999b"
server_key_iv: "6082a835521c13f47b69896f64588a01"
server_key_version: "encryptionkey@0"
I20260924 05:03:14.012820 23127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225929848825-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225929848825-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225929848825-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 05:03:14.021389 23120 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35017 } attempt: 2
I20260924 05:03:14.021852 23122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:14.021712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35017 (local address 127.0.0.1:51484)
0924 05:03:14.021771 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0924 05:03:14.021813 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260924 05:03:14.022223 23122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:14.022149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35017 (local address 127.0.0.1:51490)
0924 05:03:14.022185 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0924 05:03:14.022207 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260924 05:03:14.022326 23120 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35017: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35017: connect: Connection refused (error 111)
I20260924 05:03:14.027016 23127 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 05:03:14.028925 23127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 05:03:14.029049 23127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 05:03:14.032764 23127 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:35017
I20260924 05:03:14.032795 23194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:35017 every 8 connection(s)
I20260924 05:03:14.033129 23127 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225929848825-392-0/minicluster-data/master-2/data/info.pb
I20260924 05:03:14.034066 23195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 05:03:14.035581 23195 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36551 } has no permanent_uuid. Determining permanent_uuid...
I20260924 05:03:14.040177 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskZbcc3B/build/release/bin/kudu as pid 23127
I20260924 05:03:14.040266 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskZbcc3B/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790225929848825-392-0/minicluster-data/master-2/wal/instance
I20260924 05:03:14.040477 392 external_mini_cluster.cc:1428] Setting key 6505c86d0a8e7d64f8f1f6b47cdcb3b1
Sep 24 05:03:14 dist-test-slave-70st krb5kdc[22964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226194, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:03:14.044523 23048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:14.036236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59050 (local address 127.0.98.62:36551)
0924 05:03:14.036285 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:14.036287 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:14.036478 (+ 191us) server_negotiation.cc:407] Connection header received
0924 05:03:14.036923 (+ 445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:14.036924 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:14.036934 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:14.036976 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:03:14.037687 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:14.038178 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:14.039156 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:14.039356 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:14.039413 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:14.041707 (+ 2294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:03:14.041708 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:03:14.041709 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:03:14.041716 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:03:14.043241 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:14.043602 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:14.043603 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:14.043604 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:14.043663 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:14.043904 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:14.043904 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:14.043904 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:14.044005 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:03:14.044051 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 05:03:14.044372 (+ 321us) server_negotiation.cc:299] Negotiation successful
0924 05:03:14.044436 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 05:03:14.044696 23197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:14.036205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36551 (local address 127.0.0.1:59050)
0924 05:03:14.036359 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 05:03:14.036400 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 05:03:14.036832 (+ 432us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:03:14.037077 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:03:14.037081 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:03:14.037346 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:03:14.037576 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:03:14.037580 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:03:14.038274 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:03:14.038275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:03:14.039073 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:03:14.039075 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:03:14.039441 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:03:14.039442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:03:14.039543 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:14.040034 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:03:14.040042 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:03:14.041625 (+ 1583us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:03:14.043320 (+ 1695us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:03:14.043326 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:03:14.043328 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:03:14.043539 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:03:14.043714 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:03:14.043715 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:03:14.043715 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:03:14.043847 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:03:14.044056 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:03:14.044059 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:03:14.044246 (+ 187us) client_negotiation.cc:769] Sending connection context
0924 05:03:14.044332 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 05:03:14.044487 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":70,"threads_started":1}
I20260924 05:03:14.045521 23195 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38845 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 05:03:14 dist-test-slave-70st krb5kdc[22964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226193, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 05:03:14.049072 23045 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35017 } attempt: 2
Sep 24 05:03:14 dist-test-slave-70st krb5kdc[22964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226194, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 05:03:14.050150 23199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:14.040882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51494 (local address 127.0.98.60:35017)
0924 05:03:14.041056 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:14.041061 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:14.041085 (+ 24us) server_negotiation.cc:407] Connection header received
0924 05:03:14.041107 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:14.041109 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:14.041129 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:14.041212 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:03:14.041871 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:14.042740 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:14.043693 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:14.044086 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:14.044150 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:14.046439 (+ 2289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:03:14.046446 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:03:14.046446 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:03:14.046451 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:03:14.047783 (+ 1332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:14.048104 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:14.048108 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:14.048108 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:14.048168 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:14.048364 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:14.048364 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:14.048364 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:14.048530 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:03:14.048691 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0924 05:03:14.049979 (+ 1288us) server_negotiation.cc:299] Negotiation successful
0924 05:03:14.050051 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":76,"threads_started":1}
I20260924 05:03:14.053364 23197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:14.046140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38845 (local address 127.0.0.1:59278)
0924 05:03:14.046186 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0924 05:03:14.046197 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 05:03:14.046240 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:03:14.046367 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:03:14.046367 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:03:14.046490 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:03:14.046536 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:03:14.046537 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:03:14.047370 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:03:14.047371 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:03:14.047844 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:03:14.047846 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:03:14.048212 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:03:14.048214 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:03:14.048284 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:14.048671 (+ 387us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:03:14.048677 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:03:14.050333 (+ 1656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:03:14.052304 (+ 1971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:03:14.052305 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:03:14.052305 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:03:14.052476 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:03:14.052685 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:03:14.052685 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:03:14.052685 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:03:14.052737 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:03:14.053076 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:03:14.053077 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:03:14.053122 (+ 45us) client_negotiation.cc:769] Sending connection context
0924 05:03:14.053190 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 05:03:14.053269 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260924 05:03:14.053376 23124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:14.046202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59278 (local address 127.0.98.61:38845)
0924 05:03:14.046243 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:14.046245 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:14.046253 (+ 8us) server_negotiation.cc:407] Connection header received
0924 05:03:14.046296 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:14.046297 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:14.046305 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:14.046356 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:03:14.046612 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:14.047302 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:14.047921 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:14.048138 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:14.048200 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:14.050411 (+ 2211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:03:14.050413 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:03:14.050413 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:03:14.050420 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:03:14.052237 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:14.052563 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:14.052564 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:14.052565 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:14.052617 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:14.052818 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:14.052819 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:14.052819 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:14.053007 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:03:14.053074 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 05:03:14.053216 (+ 142us) server_negotiation.cc:299] Negotiation successful
0924 05:03:14.053291 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
Sep 24 05:03:14 dist-test-slave-70st krb5kdc[22964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226193, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 05:03:14.053781 23195 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35017 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 05:03:14 dist-test-slave-70st krb5kdc[22964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226193, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 05:03:14.056890 23047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:14.049305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35017 (local address 127.0.0.1:51510)
0924 05:03:14.049768 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0924 05:03:14.049780 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 05:03:14.049831 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:03:14.049964 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:03:14.049966 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:03:14.050093 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:03:14.050138 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:03:14.050140 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:03:14.050957 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:03:14.050958 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:03:14.051623 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:03:14.051626 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:03:14.052013 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:03:14.052014 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:03:14.052080 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:14.052457 (+ 377us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:03:14.052462 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:03:14.053792 (+ 1330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:03:14.055937 (+ 2145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:03:14.055938 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:03:14.055938 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:03:14.056136 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:03:14.056369 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:03:14.056369 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:03:14.056370 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:03:14.056425 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:03:14.056650 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:03:14.056651 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:03:14.056697 (+ 46us) client_negotiation.cc:769] Sending connection context
0924 05:03:14.056755 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 05:03:14.056825 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441}
I20260924 05:03:14.056975 23200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:14.049393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51510 (local address 127.0.98.60:35017)
0924 05:03:14.049579 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:14.049580 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:14.049818 (+ 238us) server_negotiation.cc:407] Connection header received
0924 05:03:14.049877 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:14.049878 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:14.049887 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:14.049946 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:03:14.050220 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:14.050889 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:14.051707 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:14.051921 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:14.052005 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:14.053861 (+ 1856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:03:14.053862 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:03:14.053863 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:03:14.053870 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:03:14.055861 (+ 1991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:14.056233 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:14.056234 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:14.056234 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:14.056304 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:14.056494 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:14.056495 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:14.056495 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:14.056597 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:03:14.056657 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 05:03:14.056787 (+ 130us) server_negotiation.cc:299] Negotiation successful
0924 05:03:14.056856 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":143,"threads_started":1}
Sep 24 05:03:14 dist-test-slave-70st krb5kdc[22964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226194, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 05:03:14.058923 23045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 781e6faed8424949b8f05fddf330a60b: Bootstrap starting.
I20260924 05:03:14.059404 23048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:14.051142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59060 (local address 127.0.98.62:36551)
0924 05:03:14.051175 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:14.051177 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:14.051183 (+ 6us) server_negotiation.cc:407] Connection header received
0924 05:03:14.051200 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:14.051201 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:14.051208 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:14.051251 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:03:14.051990 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:14.052554 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:14.053776 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:14.054019 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:14.054058 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:14.056271 (+ 2213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:03:14.056272 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:03:14.056273 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:03:14.056279 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:03:14.058077 (+ 1798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:14.058489 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:14.058490 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:14.058491 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:14.058543 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:14.058740 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:14.058741 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:14.058742 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:14.058872 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:03:14.058982 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 05:03:14.059242 (+ 260us) server_negotiation.cc:299] Negotiation successful
0924 05:03:14.059316 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 05:03:14.059706 23045 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 781e6faed8424949b8f05fddf330a60b: Neither blocks nor log segments found. Creating new log.
I20260924 05:03:14.060017 23045 log.cc:824] T 00000000000000000000000000000000 P 781e6faed8424949b8f05fddf330a60b: Log is configured to *not* fsync() on all Append() calls
I20260924 05:03:14.060859 23045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 781e6faed8424949b8f05fddf330a60b: No bootstrap required, opened a new log
I20260924 05:03:14.061108 23197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:14.054094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35017 (local address 127.0.0.1:51512)
0924 05:03:14.054149 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0924 05:03:14.054158 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 05:03:14.054191 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:03:14.054864 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:03:14.054865 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:03:14.055006 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:03:14.055041 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:03:14.055042 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:03:14.055878 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:03:14.055879 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:03:14.056556 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:03:14.056557 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:03:14.056999 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:03:14.057000 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:03:14.057043 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:14.057357 (+ 314us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:03:14.057362 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:03:14.058418 (+ 1056us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:03:14.060107 (+ 1689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:03:14.060107 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:03:14.060108 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:03:14.060281 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:03:14.060426 (+ 145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:03:14.060426 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:03:14.060426 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:03:14.060482 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:03:14.060772 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:03:14.060773 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:03:14.060820 (+ 47us) client_negotiation.cc:769] Sending connection context
0924 05:03:14.060908 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 05:03:14.060991 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260924 05:03:14.061173 23199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:14.054354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51512 (local address 127.0.98.60:35017)
0924 05:03:14.054762 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:14.054764 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:14.054772 (+ 8us) server_negotiation.cc:407] Connection header received
0924 05:03:14.054789 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:14.054790 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:14.054799 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:14.054855 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:03:14.055113 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:14.055796 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:14.056620 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:14.056894 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:14.056960 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:14.058607 (+ 1647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:03:14.058608 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:03:14.058608 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:03:14.058612 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:03:14.060029 (+ 1417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:14.060326 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:14.060327 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:14.060327 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:14.060374 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:14.060538 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:14.060539 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:14.060539 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:14.060676 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:03:14.060740 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 05:03:14.060957 (+ 217us) server_negotiation.cc:299] Negotiation successful
0924 05:03:14.061040 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":13,"server-negotiator.queue_time_us":391}
I20260924 05:03:14.062528 23195 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bfb729822d94bbc902c61c8d5938309: Bootstrap starting.
I20260924 05:03:14.062589 23045 raft_consensus.cc:348] T 00000000000000000000000000000000 P 781e6faed8424949b8f05fddf330a60b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "781e6faed8424949b8f05fddf330a60b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36551 } } peers { permanent_uuid: "e976476875d345c7ac7958650a004c56" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38845 } } peers { permanent_uuid: "8bfb729822d94bbc902c61c8d5938309" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35017 } }
I20260924 05:03:14.062726 23045 raft_consensus.cc:729] T 00000000000000000000000000000000 P 781e6faed8424949b8f05fddf330a60b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 781e6faed8424949b8f05fddf330a60b, State: Initialized, Role: FOLLOWER
I20260924 05:03:14.062808 23045 consensus_queue.cc:260] T 00000000000000000000000000000000 P 781e6faed8424949b8f05fddf330a60b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "781e6faed8424949b8f05fddf330a60b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36551 } } peers { permanent_uuid: "e976476875d345c7ac7958650a004c56" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38845 } } peers { permanent_uuid: "8bfb729822d94bbc902c61c8d5938309" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35017 } }
I20260924 05:03:14.063105 23045 sys_catalog.cc:564] T 00000000000000000000000000000000 P 781e6faed8424949b8f05fddf330a60b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:03:14.063078 23202 sys_catalog.cc:455] T 00000000000000000000000000000000 P 781e6faed8424949b8f05fddf330a60b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "781e6faed8424949b8f05fddf330a60b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36551 } } peers { permanent_uuid: "e976476875d345c7ac7958650a004c56" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38845 } } peers { permanent_uuid: "8bfb729822d94bbc902c61c8d5938309" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35017 } } }
I20260924 05:03:14.063166 23195 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8bfb729822d94bbc902c61c8d5938309: Neither blocks nor log segments found. Creating new log.
I20260924 05:03:14.063164 23202 sys_catalog.cc:458] T 00000000000000000000000000000000 P 781e6faed8424949b8f05fddf330a60b [sys.catalog]: This master's current role is: FOLLOWER
I20260924 05:03:14.063439 23195 log.cc:824] T 00000000000000000000000000000000 P 8bfb729822d94bbc902c61c8d5938309: Log is configured to *not* fsync() on all Append() calls
I20260924 05:03:14.064116 23195 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bfb729822d94bbc902c61c8d5938309: No bootstrap required, opened a new log
W20260924 05:03:14.064630 23214 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 781e6faed8424949b8f05fddf330a60b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 05:03:14.064720 23214 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 05:03:14.066113 23195 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8bfb729822d94bbc902c61c8d5938309 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "781e6faed8424949b8f05fddf330a60b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36551 } } peers { permanent_uuid: "e976476875d345c7ac7958650a004c56" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38845 } } peers { permanent_uuid: "8bfb729822d94bbc902c61c8d5938309" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35017 } }
I20260924 05:03:14.066256 23195 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8bfb729822d94bbc902c61c8d5938309 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bfb729822d94bbc902c61c8d5938309, State: Initialized, Role: FOLLOWER
I20260924 05:03:14.066334 23195 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8bfb729822d94bbc902c61c8d5938309 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "781e6faed8424949b8f05fddf330a60b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36551 } } peers { permanent_uuid: "e976476875d345c7ac7958650a004c56" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38845 } } peers { permanent_uuid: "8bfb729822d94bbc902c61c8d5938309" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35017 } }
I20260924 05:03:14.066630 23195 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8bfb729822d94bbc902c61c8d5938309 [sys.catalog]: configured and running, proceeding with master startup.
Sep 24 05:03:14 dist-test-slave-70st krb5kdc[22964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226193, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 05:03:14.065447 22977 master_runner.cc:186] Error getting master registration for 127.0.98.61:38845: OK, instance_id { permanent_uuid: "e976476875d345c7ac7958650a004c56" instance_seqno: 1790226193881884 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 05:03:14.068115 22977 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 05:03:14.068363 23216 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bfb729822d94bbc902c61c8d5938309 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "781e6faed8424949b8f05fddf330a60b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36551 } } peers { permanent_uuid: "e976476875d345c7ac7958650a004c56" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38845 } } peers { permanent_uuid: "8bfb729822d94bbc902c61c8d5938309" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35017 } } }
I20260924 05:03:14.068974 23216 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bfb729822d94bbc902c61c8d5938309 [sys.catalog]: This master's current role is: FOLLOWER
W20260924 05:03:14.069960 23227 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8bfb729822d94bbc902c61c8d5938309: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 05:03:14.070012 23227 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 05:03:14.071647 23124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:14.060061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59294 (local address 127.0.98.61:38845)
0924 05:03:14.060102 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:14.060103 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:14.060110 (+ 7us) server_negotiation.cc:407] Connection header received
0924 05:03:14.060186 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:14.060186 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:14.060195 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:14.060246 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:03:14.061042 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:14.061700 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:14.065229 (+ 3529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:14.065568 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:14.065613 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:14.068324 (+ 2711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:03:14.068326 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:03:14.068326 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:03:14.068333 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:03:14.069984 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:14.070528 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:14.070529 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:14.070530 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:14.070590 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:14.070778 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:14.070778 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:14.070779 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:14.070908 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:03:14.071046 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0924 05:03:14.071480 (+ 434us) server_negotiation.cc:299] Negotiation successful
0924 05:03:14.071554 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 05:03:14.070564 23127 master_runner.cc:186] Error getting master registration for 127.0.98.61:38845: OK, instance_id { permanent_uuid: "e976476875d345c7ac7958650a004c56" instance_seqno: 1790226193881884 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 05:03:14.072736 23127 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 05:03:14.073239 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 05:03:14.074311 23019 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "781e6faed8424949b8f05fddf330a60b"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59060
I20260924 05:03:14.074466 23019 raft_consensus.cc:482] T 00000000000000000000000000000000 P 781e6faed8424949b8f05fddf330a60b [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 05:03:14.074527 23019 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 781e6faed8424949b8f05fddf330a60b [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:03:14.075443 23019 raft_consensus.cc:504] T 00000000000000000000000000000000 P 781e6faed8424949b8f05fddf330a60b [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "781e6faed8424949b8f05fddf330a60b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36551 } } peers { permanent_uuid: "e976476875d345c7ac7958650a004c56" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38845 } } peers { permanent_uuid: "8bfb729822d94bbc902c61c8d5938309" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35017 } }
I20260924 05:03:14.075685 23019 leader_election.cc:288] T 00000000000000000000000000000000 P 781e6faed8424949b8f05fddf330a60b [CANDIDATE]: Term 1 election: Requested vote from peers e976476875d345c7ac7958650a004c56 (127.0.98.61:38845), 8bfb729822d94bbc902c61c8d5938309 (127.0.98.60:35017)
I20260924 05:03:14.076067 23169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "781e6faed8424949b8f05fddf330a60b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "8bfb729822d94bbc902c61c8d5938309"
I20260924 05:03:14.076174 23169 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8bfb729822d94bbc902c61c8d5938309 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:03:14.077311 23169 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8bfb729822d94bbc902c61c8d5938309 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 781e6faed8424949b8f05fddf330a60b in term 1.
I20260924 05:03:14.077590 22996 leader_election.cc:302] T 00000000000000000000000000000000 P 781e6faed8424949b8f05fddf330a60b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 781e6faed8424949b8f05fddf330a60b, 8bfb729822d94bbc902c61c8d5938309; no voters:
I20260924 05:03:14.077742 23202 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 781e6faed8424949b8f05fddf330a60b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 05:03:14.077922 23202 raft_consensus.cc:686] T 00000000000000000000000000000000 P 781e6faed8424949b8f05fddf330a60b [term 1 LEADER]: Becoming Leader. State: Replica: 781e6faed8424949b8f05fddf330a60b, State: Running, Role: LEADER
I20260924 05:03:14.078020 23202 consensus_queue.cc:237] T 00000000000000000000000000000000 P 781e6faed8424949b8f05fddf330a60b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "781e6faed8424949b8f05fddf330a60b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36551 } } peers { permanent_uuid: "e976476875d345c7ac7958650a004c56" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38845 } } peers { permanent_uuid: "8bfb729822d94bbc902c61c8d5938309" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35017 } }
I20260924 05:03:14.078554 23233 sys_catalog.cc:455] T 00000000000000000000000000000000 P 781e6faed8424949b8f05fddf330a60b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 781e6faed8424949b8f05fddf330a60b. Latest consensus state: current_term: 1 leader_uuid: "781e6faed8424949b8f05fddf330a60b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "781e6faed8424949b8f05fddf330a60b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36551 } } peers { permanent_uuid: "e976476875d345c7ac7958650a004c56" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38845 } } peers { permanent_uuid: "8bfb729822d94bbc902c61c8d5938309" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35017 } } }
I20260924 05:03:14.078670 23233 sys_catalog.cc:458] T 00000000000000000000000000000000 P 781e6faed8424949b8f05fddf330a60b [sys.catalog]: This master's current role is: LEADER
I20260924 05:03:14.085565 23235 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 05:03:14.085798 23235 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 05:03:14.093928 23048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:14.085724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59072 (local address 127.0.98.62:36551)
0924 05:03:14.085787 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:14.085789 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:14.086583 (+ 794us) server_negotiation.cc:407] Connection header received
0924 05:03:14.086673 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:14.086674 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:14.086683 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:14.086710 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:03:14.087369 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:14.088129 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:14.089140 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:14.089478 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:14.089518 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:14.091171 (+ 1653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:03:14.091172 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:03:14.091173 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:03:14.091179 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:03:14.092659 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:14.093068 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:14.093070 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:14.093070 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:14.093121 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:14.093298 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:14.093299 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:14.093299 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:14.093458 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:03:14.093538 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 05:03:14.093731 (+ 193us) server_negotiation.cc:299] Negotiation successful
0924 05:03:14.093819 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 05:03:14.095316 23169 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8bfb729822d94bbc902c61c8d5938309 [term 1 FOLLOWER]: Refusing update from remote peer 781e6faed8424949b8f05fddf330a60b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:03:14.095661 23233 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 781e6faed8424949b8f05fddf330a60b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8bfb729822d94bbc902c61c8d5938309" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 05:03:14.097599 23216 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bfb729822d94bbc902c61c8d5938309 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 781e6faed8424949b8f05fddf330a60b. Latest consensus state: current_term: 1 leader_uuid: "781e6faed8424949b8f05fddf330a60b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "781e6faed8424949b8f05fddf330a60b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36551 } } peers { permanent_uuid: "e976476875d345c7ac7958650a004c56" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38845 } } peers { permanent_uuid: "8bfb729822d94bbc902c61c8d5938309" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35017 } } }
I20260924 05:03:14.097692 23216 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bfb729822d94bbc902c61c8d5938309 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 05:03:14.099503 23124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:14.085937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59296 (local address 127.0.98.61:38845)
0924 05:03:14.085977 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:14.085978 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:14.086322 (+ 344us) server_negotiation.cc:407] Connection header received
0924 05:03:14.086467 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:14.086468 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:14.086479 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:14.086520 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:03:14.087636 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:14.088373 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:14.089904 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:14.090217 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:14.090257 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:14.095127 (+ 4870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:03:14.095128 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:03:14.095129 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:03:14.095147 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:03:14.096647 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:14.097479 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:14.097480 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:14.097480 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:14.097551 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:14.098602 (+ 1051us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:14.098603 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:14.098604 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:14.098748 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:03:14.098839 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 05:03:14.099313 (+ 474us) server_negotiation.cc:299] Negotiation successful
0924 05:03:14.099407 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 05:03:14.099581 23233 sys_catalog.cc:455] T 00000000000000000000000000000000 P 781e6faed8424949b8f05fddf330a60b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "781e6faed8424949b8f05fddf330a60b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "781e6faed8424949b8f05fddf330a60b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36551 } } peers { permanent_uuid: "e976476875d345c7ac7958650a004c56" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38845 } } peers { permanent_uuid: "8bfb729822d94bbc902c61c8d5938309" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35017 } } }
I20260924 05:03:14.099666 23233 sys_catalog.cc:458] T 00000000000000000000000000000000 P 781e6faed8424949b8f05fddf330a60b [sys.catalog]: This master's current role is: LEADER
I20260924 05:03:14.100195 23216 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bfb729822d94bbc902c61c8d5938309 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "781e6faed8424949b8f05fddf330a60b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "781e6faed8424949b8f05fddf330a60b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36551 } } peers { permanent_uuid: "e976476875d345c7ac7958650a004c56" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38845 } } peers { permanent_uuid: "8bfb729822d94bbc902c61c8d5938309" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35017 } } }
I20260924 05:03:14.100281 23216 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bfb729822d94bbc902c61c8d5938309 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 05:03:14.100620 23239 mvcc.cc:205] Tried to move back new op lower bound from 7332766491011620864 to 7332766490944364544. Current Snapshot: MvccSnapshot[applied={T|T < 7332766491011620864}]
I20260924 05:03:14.100791 23240 mvcc.cc:205] Tried to move back new op lower bound from 7332766491011620864 to 7332766490944364544. Current Snapshot: MvccSnapshot[applied={T|T < 7332766491011620864}]
I20260924 05:03:14.102209 23235 catalog_manager.cc:1347] Generated new cluster ID: aabe99969f014753a5919fb4ee5ade3c
I20260924 05:03:14.102262 23235 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 05:03:14.103943 23199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:14.086308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51520 (local address 127.0.98.60:35017)
0924 05:03:14.086349 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:14.086350 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:14.089946 (+ 3596us) server_negotiation.cc:407] Connection header received
0924 05:03:14.090060 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:14.090061 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:14.090073 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:14.090101 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:03:14.092037 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:14.092782 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:14.097967 (+ 5185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:14.098286 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:14.098324 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:14.101065 (+ 2741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:03:14.101066 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:03:14.101066 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:03:14.101074 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:03:14.102605 (+ 1531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:14.103117 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:14.103118 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:14.103119 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:14.103177 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:14.103393 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:14.103394 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:14.103394 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:14.103523 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:03:14.103615 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 05:03:14.103772 (+ 157us) server_negotiation.cc:299] Negotiation successful
0924 05:03:14.103853 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 05:03:14.113469 23120 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35017 } attempt: 3
I20260924 05:03:14.113956 23235 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 05:03:14.114786 23235 catalog_manager.cc:1504] Loading token signing keys...
Sep 24 05:03:14 dist-test-slave-70st krb5kdc[22964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790226193, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 05:03:14.123062 23122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:14.113740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35017 (local address 127.0.0.1:51534)
0924 05:03:14.113785 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 05:03:14.113796 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 05:03:14.113831 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 05:03:14.114121 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 05:03:14.114122 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 05:03:14.114302 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0924 05:03:14.114349 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:03:14.114350 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:03:14.115423 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:03:14.115424 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:03:14.116218 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 05:03:14.116220 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 05:03:14.116788 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 05:03:14.116789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 05:03:14.116858 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:14.117346 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 05:03:14.117351 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 05:03:14.119441 (+ 2090us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 05:03:14.121710 (+ 2269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:03:14.121711 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:03:14.121712 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:03:14.121937 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:03:14.122243 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 05:03:14.122244 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 05:03:14.122245 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 05:03:14.122317 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 05:03:14.122766 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 05:03:14.122767 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 05:03:14.122813 (+ 46us) client_negotiation.cc:769] Sending connection context
0924 05:03:14.122857 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 05:03:14.122956 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260924 05:03:14.123924 23199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:14.113897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51534 (local address 127.0.98.60:35017)
0924 05:03:14.113950 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:14.113952 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:14.113960 (+ 8us) server_negotiation.cc:407] Connection header received
0924 05:03:14.113981 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:14.113982 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:14.113991 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:14.114048 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:03:14.114472 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:14.115242 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:14.116323 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:14.116652 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:14.116705 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:14.119524 (+ 2819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 05:03:14.119525 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 05:03:14.119526 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 05:03:14.119533 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 05:03:14.121590 (+ 2057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:14.122066 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:14.122068 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:14.122068 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:14.122152 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 05:03:14.122395 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 05:03:14.122396 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 05:03:14.122396 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 05:03:14.122654 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 05:03:14.123566 (+ 912us) server_negotiation.cc:1035] Waiting for connection context
0924 05:03:14.123638 (+ 72us) server_negotiation.cc:299] Negotiation successful
0924 05:03:14.123775 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 05:03:14.125792 23120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e976476875d345c7ac7958650a004c56: Bootstrap starting.
I20260924 05:03:14.126559 23120 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e976476875d345c7ac7958650a004c56: Neither blocks nor log segments found. Creating new log.
I20260924 05:03:14.126843 23120 log.cc:824] T 00000000000000000000000000000000 P e976476875d345c7ac7958650a004c56: Log is configured to *not* fsync() on all Append() calls
I20260924 05:03:14.127522 23235 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 781e6faed8424949b8f05fddf330a60b: Generated new TSK 0
I20260924 05:03:14.127717 23235 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 05:03:14.131389 23120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e976476875d345c7ac7958650a004c56: No bootstrap required, opened a new log
I20260924 05:03:14.133505 23120 raft_consensus.cc:348] T 00000000000000000000000000000000 P e976476875d345c7ac7958650a004c56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "781e6faed8424949b8f05fddf330a60b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36551 } } peers { permanent_uuid: "e976476875d345c7ac7958650a004c56" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38845 } } peers { permanent_uuid: "8bfb729822d94bbc902c61c8d5938309" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35017 } }
I20260924 05:03:14.133656 23120 raft_consensus.cc:729] T 00000000000000000000000000000000 P e976476875d345c7ac7958650a004c56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e976476875d345c7ac7958650a004c56, State: Initialized, Role: FOLLOWER
I20260924 05:03:14.133729 23120 consensus_queue.cc:260] T 00000000000000000000000000000000 P e976476875d345c7ac7958650a004c56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "781e6faed8424949b8f05fddf330a60b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36551 } } peers { permanent_uuid: "e976476875d345c7ac7958650a004c56" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38845 } } peers { permanent_uuid: "8bfb729822d94bbc902c61c8d5938309" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35017 } }
I20260924 05:03:14.134071 23120 sys_catalog.cc:564] T 00000000000000000000000000000000 P e976476875d345c7ac7958650a004c56 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 05:03:14.134634 23094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "781e6faed8424949b8f05fddf330a60b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "e976476875d345c7ac7958650a004c56"
I20260924 05:03:14.134748 23094 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e976476875d345c7ac7958650a004c56 [term 0 FOLLOWER]: Advancing to term 1
I20260924 05:03:14.135658 23094 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e976476875d345c7ac7958650a004c56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 781e6faed8424949b8f05fddf330a60b in term 1.
I20260924 05:03:14.136231 23093 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e976476875d345c7ac7958650a004c56 [term 1 FOLLOWER]: Refusing update from remote peer 781e6faed8424949b8f05fddf330a60b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 05:03:14.136494 23202 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 781e6faed8424949b8f05fddf330a60b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e976476875d345c7ac7958650a004c56" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 05:03:14.137487 23257 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e976476875d345c7ac7958650a004c56: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 05:03:14.137547 23257 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 05:03:14.138772 23246 sys_catalog.cc:455] T 00000000000000000000000000000000 P e976476875d345c7ac7958650a004c56 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "781e6faed8424949b8f05fddf330a60b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "781e6faed8424949b8f05fddf330a60b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36551 } } peers { permanent_uuid: "e976476875d345c7ac7958650a004c56" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38845 } } peers { permanent_uuid: "8bfb729822d94bbc902c61c8d5938309" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35017 } } }
I20260924 05:03:14.138868 23246 sys_catalog.cc:458] T 00000000000000000000000000000000 P e976476875d345c7ac7958650a004c56 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 05:03:14.138934 23246 sys_catalog.cc:455] T 00000000000000000000000000000000 P e976476875d345c7ac7958650a004c56 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "781e6faed8424949b8f05fddf330a60b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "781e6faed8424949b8f05fddf330a60b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36551 } } peers { permanent_uuid: "e976476875d345c7ac7958650a004c56" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38845 } } peers { permanent_uuid: "8bfb729822d94bbc902c61c8d5938309" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35017 } } }
I20260924 05:03:14.138979 23246 sys_catalog.cc:458] T 00000000000000000000000000000000 P e976476875d345c7ac7958650a004c56 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 05:03:14.139751 23262 sys_catalog.cc:455] T 00000000000000000000000000000000 P e976476875d345c7ac7958650a004c56 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 781e6faed8424949b8f05fddf330a60b. Latest consensus state: current_term: 1 leader_uuid: "781e6faed8424949b8f05fddf330a60b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "781e6faed8424949b8f05fddf330a60b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36551 } } peers { permanent_uuid: "e976476875d345c7ac7958650a004c56" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38845 } } peers { permanent_uuid: "8bfb729822d94bbc902c61c8d5938309" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35017 } } }
I20260924 05:03:14.139834 23262 sys_catalog.cc:458] T 00000000000000000000000000000000 P e976476875d345c7ac7958650a004c56 [sys.catalog]: This master's current role is: FOLLOWER
W20260924 05:03:14.148037 23124 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:03:14.147029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59304 (local address 127.0.98.61:38845)
0924 05:03:14.147072 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:14.147074 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:14.147439 (+ 365us) server_negotiation.cc:407] Connection header received
0924 05:03:14.147524 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:14.147525 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:14.147536 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:14.147563 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:03:14.147971 (+ 408us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59304: BlockingRecv error: recv got EOF from 127.0.0.1:59304 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260924 05:03:14.148037 23048 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:03:14.147206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59074 (local address 127.0.98.62:36551)
0924 05:03:14.147242 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:14.147244 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:14.147337 (+ 93us) server_negotiation.cc:407] Connection header received
0924 05:03:14.147363 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:14.147364 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:14.147376 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:14.147421 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:03:14.147971 (+ 550us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59074: BlockingRecv error: recv got EOF from 127.0.0.1:59074 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260924 05:03:14.148756 23199 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:03:14.147510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51540 (local address 127.0.98.60:35017)
0924 05:03:14.147549 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:14.147551 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:14.148307 (+ 756us) server_negotiation.cc:407] Connection header received
0924 05:03:14.148400 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:14.148401 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:14.148411 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:14.148456 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:03:14.148695 (+ 239us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51540: BlockingRecv error: recv got EOF from 127.0.0.1:51540 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260924 05:03:14.150205 23124 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:03:14.149663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59312 (local address 127.0.98.61:38845)
0924 05:03:14.149697 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:14.149698 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:14.149704 (+ 6us) server_negotiation.cc:407] Connection header received
0924 05:03:14.149734 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:14.149734 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:14.149750 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:14.149792 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:03:14.150158 (+ 366us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59312: BlockingRecv error: recv got EOF from 127.0.0.1:59312 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260924 05:03:14.152482 23124 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:03:14.152050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59326 (local address 127.0.98.61:38845)
0924 05:03:14.152079 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:14.152081 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:14.152087 (+ 6us) server_negotiation.cc:407] Connection header received
0924 05:03:14.152129 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:14.152130 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:14.152138 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:14.152182 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:03:14.152441 (+ 259us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59326: BlockingRecv error: recv got EOF from 127.0.0.1:59326 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260924 05:03:14.156057 23124 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:03:14.155304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59340 (local address 127.0.98.61:38845)
0924 05:03:14.155352 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:14.155353 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:14.155612 (+ 259us) server_negotiation.cc:407] Connection header received
0924 05:03:14.155698 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:14.155699 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:14.155708 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:14.155734 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:03:14.156014 (+ 280us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59340: BlockingRecv error: recv got EOF from 127.0.0.1:59340 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
W20260924 05:03:14.162283 23124 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:03:14.161479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59346 (local address 127.0.98.61:38845)
0924 05:03:14.161518 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:14.161520 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:14.161762 (+ 242us) server_negotiation.cc:407] Connection header received
0924 05:03:14.161852 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:14.161854 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:14.161865 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:14.161891 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:03:14.162219 (+ 328us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59346: BlockingRecv error: recv got EOF from 127.0.0.1:59346 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260924 05:03:14.172679 23124 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:03:14.171729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59356 (local address 127.0.98.61:38845)
0924 05:03:14.171772 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:14.171775 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:14.172054 (+ 279us) server_negotiation.cc:407] Connection header received
0924 05:03:14.172159 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:14.172160 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:14.172172 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:14.172202 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:03:14.172617 (+ 415us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59356: BlockingRecv error: recv got EOF from 127.0.0.1:59356 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260924 05:03:14.191197 23124 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:03:14.190238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59366 (local address 127.0.98.61:38845)
0924 05:03:14.190284 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:14.190286 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:14.190593 (+ 307us) server_negotiation.cc:407] Connection header received
0924 05:03:14.190700 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:14.190701 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:14.190715 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:14.190746 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:03:14.191137 (+ 391us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59366: BlockingRecv error: recv got EOF from 127.0.0.1:59366 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260924 05:03:14.225641 23124 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:03:14.224695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59378 (local address 127.0.98.61:38845)
0924 05:03:14.224740 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:14.224742 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:14.225002 (+ 260us) server_negotiation.cc:407] Connection header received
0924 05:03:14.225101 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:14.225102 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:14.225115 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:14.225144 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:03:14.225579 (+ 435us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59378: BlockingRecv error: recv got EOF from 127.0.0.1:59378 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260924 05:03:14.291718 23124 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:03:14.291103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59394 (local address 127.0.98.61:38845)
0924 05:03:14.291158 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:14.291160 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:14.291199 (+ 39us) server_negotiation.cc:407] Connection header received
0924 05:03:14.291281 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:14.291282 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:14.291293 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:14.291341 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:03:14.291659 (+ 318us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59394: BlockingRecv error: recv got EOF from 127.0.0.1:59394 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
W20260924 05:03:14.421568 23124 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:03:14.420857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59400 (local address 127.0.98.61:38845)
0924 05:03:14.420937 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:14.420939 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:14.421022 (+ 83us) server_negotiation.cc:407] Connection header received
0924 05:03:14.421055 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:14.421056 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:14.421068 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:14.421142 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:03:14.421504 (+ 362us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59400: BlockingRecv error: recv got EOF from 127.0.0.1:59400 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 05:03:14.677155 23202 sys_catalog.cc:455] T 00000000000000000000000000000000 P 781e6faed8424949b8f05fddf330a60b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "781e6faed8424949b8f05fddf330a60b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "781e6faed8424949b8f05fddf330a60b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36551 } } peers { permanent_uuid: "e976476875d345c7ac7958650a004c56" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38845 } } peers { permanent_uuid: "8bfb729822d94bbc902c61c8d5938309" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35017 } } }
I20260924 05:03:14.677274 23202 sys_catalog.cc:458] T 00000000000000000000000000000000 P 781e6faed8424949b8f05fddf330a60b [sys.catalog]: This master's current role is: LEADER
W20260924 05:03:14.679072 23124 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 05:03:14.678480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59416 (local address 127.0.98.61:38845)
0924 05:03:14.678556 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:14.678558 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:14.678568 (+ 10us) server_negotiation.cc:407] Connection header received
0924 05:03:14.678638 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:14.678638 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:14.678649 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:14.678696 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 05:03:14.679018 (+ 322us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59416: BlockingRecv error: recv got EOF from 127.0.0.1:59416 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 05:03:15.070401 23227 catalog_manager.cc:1259] Loaded cluster ID: aabe99969f014753a5919fb4ee5ade3c
I20260924 05:03:15.070472 23227 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8bfb729822d94bbc902c61c8d5938309: loading cluster ID for follower catalog manager: success
I20260924 05:03:15.071261 23227 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8bfb729822d94bbc902c61c8d5938309: acquiring CA information for follower catalog manager: success
I20260924 05:03:15.071555 23227 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8bfb729822d94bbc902c61c8d5938309: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 05:03:15.137969 23257 catalog_manager.cc:1259] Loaded cluster ID: aabe99969f014753a5919fb4ee5ade3c
I20260924 05:03:15.138047 23257 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e976476875d345c7ac7958650a004c56: loading cluster ID for follower catalog manager: success
I20260924 05:03:15.138854 23257 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e976476875d345c7ac7958650a004c56: acquiring CA information for follower catalog manager: success
I20260924 05:03:15.139138 23257 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e976476875d345c7ac7958650a004c56: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 05:03:15.194845 23330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:15.192206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59430 (local address 127.0.98.61:38845)
0924 05:03:15.192309 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:15.192311 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:15.192369 (+ 58us) server_negotiation.cc:407] Connection header received
0924 05:03:15.192442 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:15.192442 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:15.192510 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:15.192576 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:03:15.192739 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:15.193284 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:15.194089 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:15.194298 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:15.194352 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:15.194502 (+ 150us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:03:15.194617 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:03:15.194686 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 05:03:15.194724 (+ 38us) server_negotiation.cc:299] Negotiation successful
0924 05:03:15.194746 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82,"thread_start_us":50,"threads_started":1}
I20260924 05:03:15.198638 23336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:15.196416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51546 (local address 127.0.98.60:35017)
0924 05:03:15.196511 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:15.196512 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:15.196519 (+ 7us) server_negotiation.cc:407] Connection header received
0924 05:03:15.196552 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:15.196552 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:15.196585 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:15.196650 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:03:15.196720 (+ 70us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:15.197179 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:15.197901 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:15.198096 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:15.198172 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:15.198294 (+ 122us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:03:15.198395 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:03:15.198456 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 05:03:15.198536 (+ 80us) server_negotiation.cc:299] Negotiation successful
0924 05:03:15.198555 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79,"thread_start_us":53,"threads_started":1}
I20260924 05:03:15.202764 23330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:15.200235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59436 (local address 127.0.98.61:38845)
0924 05:03:15.200282 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:15.200283 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:15.200420 (+ 137us) server_negotiation.cc:407] Connection header received
0924 05:03:15.200466 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:15.200466 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:15.200477 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:15.200519 (+ 42us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:03:15.200597 (+ 78us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:15.201310 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:15.202161 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:15.202364 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:15.202417 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:15.202537 (+ 120us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:03:15.202572 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:03:15.202617 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 05:03:15.202659 (+ 42us) server_negotiation.cc:299] Negotiation successful
0924 05:03:15.202681 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 05:03:15.203289 23336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:15.200242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51560 (local address 127.0.98.60:35017)
0924 05:03:15.200331 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:15.200333 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:15.200682 (+ 349us) server_negotiation.cc:407] Connection header received
0924 05:03:15.200761 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:15.200762 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:15.200771 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:15.200813 (+ 42us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:03:15.200902 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:15.201386 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:15.202584 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:15.202865 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:15.202917 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:15.203060 (+ 143us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:03:15.203093 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:03:15.203136 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 05:03:15.203193 (+ 57us) server_negotiation.cc:299] Negotiation successful
0924 05:03:15.203212 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 05:03:15.203352 23343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 05:03:15.200295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59084 (local address 127.0.98.62:36551)
0924 05:03:15.200399 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0924 05:03:15.200401 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 05:03:15.200409 (+ 8us) server_negotiation.cc:407] Connection header received
0924 05:03:15.200448 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 05:03:15.200448 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 05:03:15.200487 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 05:03:15.200562 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 05:03:15.200703 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:15.201476 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:15.202487 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 05:03:15.202769 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 05:03:15.202825 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 05:03:15.203013 (+ 188us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 05:03:15.203119 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 05:03:15.203155 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0924 05:03:15.203250 (+ 95us) server_negotiation.cc:299] Negotiation successful
0924 05:03:15.203272 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86,"thread_start_us":55,"threads_started":1}
I20260924 05:03:15.204898 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 22977
I20260924 05:03:15.209487 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 23052
I20260924 05:03:15.213536 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskZbcc3B/build/release/bin/kudu with pid 23127
2026-09-24T05:03:15Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2675 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5378 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (265369 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST